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ABSTRACT 


The  primary  objective  of  health  care  should  be  to  achieve  good  or  improved 
outcomes  for  patients  in  a  cost  effective  manner  (Agency  for  Health  Care  Policy  and 
Research  1995b).  This  realization  has  prompted  the  health  care  community  to  develop 
new  methods  of  accurately  calculating  and  measuring  health  care  cost  and  quality.  As 
researchers  seek  new  methods  to  better  quantify  the  value  equation,  the  Army  Medical 
Command  (MEDCOM)  is  also  being  asked  to  justify  health  care  related  expenses  and 
demonstrate  quality  outcomes  as  the  command  transitions  to  a  managed  care 
environment.  To  improve  the  processes  of  care  provided  in  its  Medical  Treatment 
Facilities  (MTFs)  from  both  cost  and  quality  perspectives,  the  MEDCOM  must  exploit 
effective  utilization  management  techniques  whenever  possible  by  identifying  benchmark 
clinical  procedures  and  processes  and  adopting  these  best  practices  at  other  MEDCOM 
facilities. 

The  clinical  benchmarking  model  developed  in  this  project  allows  MEDCOM 
decision  makers  to  ensure  the  Military  Health  Service  System  (MHSS)  remains  the  health 
plan  of  choice  for  military  beneficiaries  and  assists  with  utilization  management  (UM)  by 
providing  efficient  benchmarks  for  comparison.  The  greatest  benefit  from  this  project 
will  come  from  the  ability  to  objectively  demonstrate,  through  cost  and  quality  outcomes 
measures,  that  the  MHSS  provides  better  health  care  value  than  civilian  competitors  and 
that  it  remains  the  best  health  care  choice  for  beneficiaries. 
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CHAPTER  1 


INTRODUCTION 

While  the  primary  objective  of  health  care  should  be  to  achieve  good  or  improved 
outcomes  for  patients  in  a  cost  effective  manner  (Agency  for  Health  Care  Policy  and 
Research  1995b),  Americans  are  finding  that  quality  of  care  is  not  a  given.  This 
realization  prompted  a  movement  toward  public  accountability  by  health  care  providers 
for  outcomes  of  care.  Accountability  requires  measurement  of  the  quality  of  care 
provided  to  populations  of  patients  and  policies  that  promote  continuous  improvement  of 
quality  of  care.  In  concert  with  this  new  accountability,  recent  changes  in  the  political 
environment  have  put  the  civilian  health  care  community  under  increasing  pressure  to 
become  more  efficient  and  reduce  costs.  The  relative  improvement  in  health  care 
outcomes  compared  to  the  cost  of  the  health  care  services  used  to  achieve  them  are  the 
components  of  the  health  care  value  equation.  Stated  another  way,  health  care  value  is 
high  quality  care  provided  cost  efficiently  (Manus,  Wemer  and  Strub  1994).  Increasing 
competition  among  health  care  providers  and  the  spread  of  managed  care  have  required 
improved  methods  of  defining  and  measuring  health  care  value.  In  response  to  the 
changing  environment,  the  health  care  provider  community  has  developed  new  methods 
of  accurately  calculating  and  measuring  health  care  cost  and  quality. 

The  Military  Health  Service  System  (MHSS)  has  also  experienced  similar 
pressure  to  become  more  efficient.  As  researchers  seek  new  methods  to  better  quantify 
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the  value  equation,  the  Army  Medical  Command  (MEDCOM)  is  also  being  asked  to 
justify  health  care  related  expenses  and  demonstrate  quality  outcomes  as  the  command 
transitions  to  a  managed  care  environment.  The  result  is  that  the  MEDCOM  is  now, 
more  than  ever,  accountable  to  Congress  and  the  American  people  for  not  only  the  cost  of 
military  health  care,  but  for  the  efficiency,  quality  and  value  as  well. 

Background 

Several  factors  have  influenced  the  transition  to  managed  care  at  the  MEDCOM. 
Factors  such  as  the  national  health  care  reform  initiative,  continuing  military  strength 
reductions  and  the  desire  at  all  levels  of  government  to  improve  efficiency  have 
combined  to  create  a  strong  impetus  for  change  in  the  MEDCOM.  The  result  is  that  the 
MEDCOM  must  now  examine  how  it  provides  care  in  its  Medical  Treatment  Facilities 
(MTFs)  and  determine  how  the  care  delivery  processes  can  be  improved  from  both  cost 
and  quality  perspectives. 

In  response  to  the  changing  environment,  the  MEDCOM  began  to  improve  the 
efficiency  of  the  care  delivery  process  through  establishment  of  regional  referral 
networks.  These  referral  networks  were  the  first  step  in  transforming  the  MEDCOM  into 
a  managed  care  network  instead  of  a  loose  affiliation  of  hospitals.  As  the  other  military 
services  began  to  develop  similar  health  care  referral  systems,  geographical  and  mission 
similarities  prompted  the  decision  to  merge  the  health  care  facilities  of  the  three  services 
into  one  managed  care  system. 
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As  part  of  the  tri-service  health  care  reorganization,  the  Office  of  the  Assistant 
Secretary  of  Defense  for  Health  Affairs  (OASD(HA))  was  given  responsibility  for 
oversight  of  military  health  care  operations  in  the  Continental  United  States  (CONUS) 
and  Europe.  Although  the  MTFs  were  still  under  the  operational  control  of  the  three 
military  services,  OASD(HA)  implemented  programs  designed  to  coordinate  and  oversee 
a  comprehensive,  coordinated  health  care  benefit  for  active  duty  military,  their 
dependents,  retirees  and  other  beneficiaries.  This  coordination  was  accomplished  by 
organizing  the  MTFs  of  the  three  services  into  one  Military  Health  Services  System 
(MHSS)  divided  into  twelve  geographic  health  care  regions  (Figure  1).  Each  region  is 
managed  by  a  Department  of  Defense  (DoD)  Lead  Agent  responsible  for  coordinating  all 
health  care  delivery  to  military  beneficiaries  within  their  region. 

To  assist  Lead  Agents  in  managing  care  provided  outside  the  MTFs  by  civilian 
providers,  OASD(HA)  developed  and  awarded  contracts  to  augment  MTF  care  by 
providing  services  not  available  at  the  MTF.  As  the  contracts  were  awarded,  only 
rudimentary  mechanisms  were  in  place  to  evaluate  the  quality  and  efficiency  of  care 
provided  by  the  contracted  providers.  As  MHSS  resources  became  more  constrained  and 
MTF  efficiency  became  more  critical,  Lead  Agents  and  Hospital  Commanders  were 
required  to  evaluate  MTF  operations  against  the  civilian  contract  providers  to  determine 
which  provided  the  better  value.  This  became  a  difficult  process  due  to  the  lack  of  good 
health  care  value  measurement  tools  available  to  assist  in  the  decision  process. 

During  this  same  period,  the  MEDCOM  organization  underwent  one  of  the  most 
extensive  reorganizations  since  the  1940s.  Many  factors  influenced  the  need  to 
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TRICARE  HEALTH  SERVICE  REGIONS 


REGION  8 
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REGION  5 

WRIGHT  PATTERSON 
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WILFORD  HALL 


REGION 
WALTER  REED 


12  REGIONS 
LEAD  AGENT 


Hurt  and  Cyr  1996 


Figure  1.  DoD  TRICARE  Regions 


reorganize,  such  as  the  reduced  need  for  a  large  military  force  and  Army  downsizing, 
fundamental  changes  in  the  conduct  of  military  operations,  changing  battlefield 
scenarios,  increased  lethality,  increased  technology  and  a  shift  to  a  power  projection 
platform  strategy.  Internal  to  the  MEDCOM,  the  previous  structure  did  not  adequately 
identify  the  types  of  work  to  be  done  at  each  level  of  the  organization,  confusing 
strategic,  operational,  and  tactical  roles.  The  intent  of  the  reorganization  effort  was  to 
separate  the  organization  into  clearly  defined  strategic,  operational,  and  tactical  levels  to 
enhance  efficiency  and  eliminate  redundancy,  and  to  reengineer  the  organization  to  be 
more  responsive  to  the  changing  environment. 
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In  July  1993,  Army  MTFs,  their  command  entities  and  other  associated  agencies 
and  activities  were,  at  the  direction  of  the  Army  Surgeon  General,  restructured  in  to  what 
is  presently  known  as  the  Army  Medical  Command  (Figure  2). 


Hurt  and  Cyr  1996 

Figure  2.  AMEDD  Reorganization  Model 


The  new  MEDCOM  organization  consists  of  37  Medical  Treatment  Facilities  (MTFs) 
organized  into  seven  Health  Service  Support  Areas  (HSSAs)  or  regions  located  in 
CONUS  and  Europe.  In  addition  to  the  seven  HSSAs,  the  MEDCOM  has  five  other 
Major  Subordinate  Commands  (MSCs),  responsible  for  medical  research  and  medical 
materiel  development,  health  care  promotion  and  preventive  medicine,  food  inspection 
and  animal  health  issues,  education  and  doctrine  development,  and  dental  health  of 
soldiers  (Figure  3).  The  MEDCOM  is  no  longer  subordinate  to  the  Office  of  the  Surgeon 
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General  (OTSG).  which  is  now  responsible  for  providing  liaison  with  policy  makers  in 
the  National  Capitol  Region  and  for  communicating  Army  Medical  Department 
(AMEDD)  policy  lo  other  Department  of  the  Army  staff. 


MEDCOM 


Him  unci  Cyr  IWft 

Figure  3.  MEDCOM  Organizational  Chart 


The  MEDCOM  has  assumed  the  strategic  management  role  for  the  AMEDD, 
passing  operational  level  functions  previously  done  at  Health  Services  Command  (HSC) 
down  to  the  newly  created  HSSAs.  These  organizational  changes  empowered  the  newly 
appointed  HSSA  commanders  to  efficiently  manage  their  regions,  and  for  the  first  time 
held  them  accountable  for  health  care  delivery  and  readiness  operations  of  Army  MTFs 
located  within  their  regions.  As  the  HSSAs  assumed  the  operational  responsibility  to 
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employ  health  care  assets  efficiently,  few  tools  existed  to  assist  them  in  managing  their 
assets. 

During  the  period  of  AMEDD  reorganization  and  the  transition  to  TRICARE,  the 
Department  of  Defense  (DoD)  Inspector  General  (IG)  conducted  an  audit  of  the  current 
state  of  utilization  management  (UM)  in  DoD  MTFs.  Results  of  the  audit  indicated  that 
while  many  opportunities  existed  to  manage  more  efficiently,  few  MTFs  had  put  effective 
UM  measures  in  place  (Department  of  Defense  Inspector  General  1995).  Subsequent 
analysis  conducted  by  OASD(HA),  the  DoD  Program  Analysis  and  Evaluation  (PAE) 
office,  and  the  Congressional  Budget  Office  (CBO)  confirmed  these  findings  to  varying 
degrees  (Office  of  the  Under  Secretary  of  Defense  (Comptroller)  1995).  As  a  result  of 
these  studies,  the  MEDCOM  received  a  directive  from  OASD(HA),  to  develop  UM 
policies  and  practices  for  MEDCOM  MTFs  (Office  of  the  Assistant  Secretary  of  Defense 
for  Health  Affairs  1995a).  OASD(HA)  also  organized  working  groups  to  address  specific 
issues  identified  by  the  DoDIG  Report. 

To  further  incentivize  the  MTFs  to  implement  effective  UM  measures, 
OASD(HA)  incorporated  a  UM  decrement  as  part  of  the  Defense  Health  Plan  (DHP) 
budget  process.  The  proposed  OASD(HA)  decrement  was  intended  to  be  progressive, 
reducing  the  DHP  budget  one  percent  in  Fiscal  Year  1997,  increasing  to  four  percent  in 
1999  (Office  of  the  Under  Secretary  of  Defense  (Comptroller)  1995).  Following 
submission  of  the  budget  plan,  the  DoD  Comptroller  modified  the  proposal  to  increase 
the  decrement  to  five  percent  per  year  for  four  years  beginning  in  Fiscal  Year  1997 
(Office  of  the  Under  Secretary  of  Defense  (Comptroller)  1995).  This  threat  of  significant 
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budget  reductions  added  an  increased  sense  of  urgency  to  the  MEDCOM  Headquarters 
staff  as  they  began  the  process  of  developing  effective  measures  of  health  care  quality, 
efficiency,  and  value. 


Conditions  Which  Prompted  the  Study 

Historically,  MEDCOM  performance  measures  have  monitored  aspects  of  the 
organization  related  to  cost  and  workload  (Appendix  1).  With  the  changing  environment 
and  organizational  mission,  cost  and  workload  performance  measures  are  of  decreasing 
value.  As  the  MEDCOM,  like  the  rest  of  the  Federal  Government,  finds  that  resources 
continue  to  shrink  and  demand  for  quality  products  and  services  continues  to  grow 
(Financial  Management  Service  1993),  structure  oriented  cost  and  workload  performance 
measures  do  not  provide  information  adequate  to  efficiently  manage  military  health  care 
organizations. 

Environmental  and  organizational  changes  have  created  turbulence  within  the 
MEDCOM  Headquarters  and  MSCs  as  they  attempt  to  redefine  their  roles  and  missions, 
evaluate  their  performance,  and  meet  the  expectations  of  their  customers.  Although  the 
MEDCOM  and  MHSS  organizations  are  evolving  to  keep  pace  with  the  environment,  the 
current  performance  indicators  are  not.  At  the  MEDCOM  headquarters  level,  the  need  to 
develop  strategic  level  performance  measures  accurately  depicting  the  relationship 
between  health  care  cost,  quality  and  value  provided  by  the  HSSAs  and  civilian  sector 
contracted  providers  prompted  this  study.  The  current  quality  management  indicators  do 
not  accurately  measure  progress  toward  attainment  of  key  MEDCOM  and  MHSS 
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strategic  goals,  and  the  indicators  are  not  strategic  in  scope  as  they  should  be  under  the 
reorganization. 

In  September  1995  the  MEDCOM  staff  began  evaluating  potential  quality 
management  indicators.  The  staff  sought  indicators  which  would  measure  progress 
toward  attainment  of  the  strategic  goals  of  the  new  MEDCOM  organization,  would 
facilitate  comparisons  with  civilian  organizations,  would  meet  OASD(HA)  guidelines  for 
developing  new  indicators  (Table  1)  (Office  of  the  Assistant  Secretary  of  Defense  for 
Health  Affairs  1995a),  and  would  be  compatible  with  existing  data  systems. 


Table  1.  DoD  Health  Affairs  Clinical  Performance  Indicator  Requirements 


DoD  Health  Affairs  Clinical  Performance  Indicator 
Requirements 

Measure  Both  Process  and  Outcome 

Address  High  Risk,  High  Volume,  and/or  Problem  Prone  Areas 

Address  Both  Clinical  Quality  and  Service  Issues 

Are  Systemic  in  Design 

Are  Carried  Out  Uniformly  Across  Components 

Have  Appropriate  Breadth  and  Frequency 

Include  Areas  of  Prevention,  Physiological  Function,  Functional 
Status,  Physical  and  Psychological  Comfort 

Provide  Data  Useful  for  Continuous  Measurement  and  Assessment 

Impact  on  Quality  Outcome 

Impact  on  Patient  Satisfaction 

Rigor  of  Development  and  Testing 

Ease  of  Implementation 

Vulnerability  (Resistance)  to  “Gaming” 

Interpretability  as  a  Measure  of  Quality 

Office  of  the  Assistant  Secretary  of  Defense  for  Health  Affairs  1995a 
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In  defining  new  performance  measures,  the  staff  had  difficulty  identifying  measures 
which  would  accurately  measure  progress  toward  the  goals  of  becoming  the  managed 
care  system  of  choice  and  becoming  a  learning  organization  that  were  obtainable  through 
existing  data  systems.  To  develop  these  measures,  and  to  adhere  to  OASD(HA) 
guidelines,  the  staff  began  to  review  measures  currently  in  use  in  the  civilian  health  care 
sector  that  measure  key  clinical  processes  and  outcomes,  and  that  were  compatible  with 
existing  information  systems.  When  evaluating  new  performance  measures,  the 
MEDCOM  staff  searched  for  methods  of  measuring  not  just  the  cost  of  health  care,  but 
health  care  quality  and  value  as  well.  Applying  the  value  definition  of  best  quality  for  the 
most  reasonable  cost  (Manus,  Werner  and  Strub  1994),  the  staff  determined  that  the 
MEDCOM  needed  to  develop  better  methods  of  measuring  health  care  quality  and  cost 
for  specific  key  procedures  and  product  lines. 

Statement  of  the  Problem 

Performance  measures  currently  in  use  at  the  MEDCOM  do  not  provide  the 
information  required  to  evaluate  progress  toward  the  Health  Affairs  and  MEDCOM 
strategic  goals,  do  not  measure  UM  and  health  care  value,  do  not  facilitate  effective 
collaboration  and  comparisons  with  civilian  sector  health  care  benchmark  organizations, 
and  do  not  meet  OASD(HA)  requirements  for  health  care  clinical  performance  measures. 
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Literature  Review 


Performance  Measures 

Historically,  health  care  performance  measures  have  been  retrospective,  lacking 
the  “predictive  ability”  sought  by  industry  leaders  today  (Birchard  1995).  The 
MEDCOM  is  not  alone  in  its  quest  for  better  performance  measures.  As  many  as  80 
percent  of  American  businesses  are  currently  exploring  ways  to  change  their  performance 
measurement  systems  to  better  reflect  the  changing  goals  of  the  organization  (Birchard 
1995).  This  movement  toward  new  measures  in  industry  is  a  result  of  the  tremendous 
effort  expended  in  reorganizing  and  re-engineering  organizations,  both  within 
government  and  in  the  civilian  sector,  to  provide  products  and  services  in  the  most 
efficient  manner  possible  (Birchard  1995).  Restructured  businesses  and  organizations 
must  now  find  new  measurement  tools  to  evaluate  progress  toward  revised  corporate 
goals. 

Performance  measures  can  be  constructed  to  measure  a  variety  of  attributes  of  an 
organization  (Appendix  2).  The  purpose  of  management  indicators  is  to  objectively 
monitor  how  well  an  organization  performs  its  core  business  processes  and  accomplishes 
its  strategic  objectives  through  the  delivery  of  products,  and  services,  and  how  efficiently 
it  manages  the  essential  processes  of  delivering  those  products  and  services  (Financial 
Management  Services  1993).  These  measures  can  then  be  used  to  refine  decision 
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making,  to  facilitate  process  improvement,  and  to  evaluate  effects  of  previous  changes 
(O’Leary  1993). 

Performance  measures  should  be  meaningful  to  managers  and  customers,  should  be 
simple,  logical,  repeatable  and  timely;  should  show  a  trend,  be  clearly  defined,  be 
economical  to  collect  and  tell  how  well  an  organization  meets  its  goals  and  objectives 
(Office  of  the  Assistant  Secretary  of  Defense  for  Health  Affairs  1995b).  When  defining 
performance  measures  it  is  important  to  think  in  terms  of  stakeholder  and  customer  needs 
and  desires,  as  well  as  organizational  needs,  and  to  evaluate  their  satisfaction  with  your 
efforts  (Financial  Management  Service  1993).  Developed  properly,  performance  measures 
can  be  a  valuable  tool  for  organizational  self-assessment,  goal  setting,  and  progress 
monitoring  (Financial  Management  Service  1993). 

Performance  measures  provide  information  regarding  progress  toward 
organizational  goals.  To  be  effective,  performance  measures  must  be  tied  to  strategic  goals 
at  all  levels  of  the  organization,  corresponding  to  the  organizational  level  which  controls 
that  particular  aspect  of  the  organization  (Financial  Management  Service  1993). 
Performance  measures  should  be  limited  to  the  minimum  number  necessary  to  measure 
progress  toward  organizational  goals  at  each  level  of  the  organization,  with  measures  and 
goals  at  each  level  supporting  measures  and  goals  of  the  next  higher  level  (Figure  4). 
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Figure  4.  Strategic  Planning  Model 


Quality  Measures 

Performance  measures  have  become  increasingly  important  to  health  care 
organizations  seeking  to  improve  quality,  ensure  accountability,  formulate  policy,  plan  and 
budget,  and  assure  equitable  distribution  of  services  (Harris  1994).  As  the  American 
health  care  industry  has  embraced  the  teaching  of  Demming,  Juran  and  others  and  begun 
practicing  Total  Quality  Management  (TQM),  whole  sectors  of  the  industry  have  begun  to 
develop  better  process  and  outcomes  related  performance  measures.  For  the  health  care 
industry,  the  shift  to  process  and  outcomes  related  performance  measures  is  as  much  a 
competitive  necessity  as  a  quest  for  quality  (Harris  1994).  As  managed  care  plans  become 
more  efficient  and  as  price  and  cost  become  more  standardized  throughout  the  industry, 
civilian  health  care  plans  are  increasingly  being  evaluated  by  customer  and  payors  on  their 
value  and  outcomes  quality  and  not  just  low  cost  (Hale  and  Weiner  1994). 
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Quality  measurement  has  become  a  critical  issue  in  almost  all  sectors  of  business 
and  industry.  It  is  now  recognized  that  error,  waste,  and  rework  lead  to  higher  costs  and 
less  customer  satisfaction.  This  recognition  has  prompted  a  new  interest  in  formally 
adopting  quality  improvement  and  management  techniques  in  industry  in  general  and  in 
the  health  care  sector  in  particular  (Berwick  1989).  This  is  especially  true  as  health  care 
organizations  attempt  to  become  more  competitive  and  seek  better  ways  to  measure  not 
just  cost,  but  quality  and  value  as  well. 

At  the  heart  of  the  quality  improvement  methodology  is  the  quality  management 
cycle  (Figure  5)  or  Shewart  cycle.  In  its  simplest  form  this  cycle  consists  of  planning  a 
new  or  improved  process  that  is  expected  to  yield  an  improved  outcome,  doing  or 
implementing  the  process,  checking  to  determine  if  the  process  is  working  as  planned  and 
is  having  the  expected  outcomes,  and  then  acting  on  the  information  that  has  been  obtained 
to  reassess  the  plan  or  continue  implementing  it  (Agency  for  Health  Care  Policy  and 
Research  1995b). 

In  assessing  and  monitoring  the  quality  of  health  care,  the  health  care  industry  is 
using  TQM  principles  to  move  toward  a  broader  population  based  perspective  of  health 
care  quality.  The  roots  of  population  based  health  care  quality  improvement  and 
measurement  extend  to  the  time  of  the  Crimean  War  and  Florence  Nightingale’s 
introduction  of  nutrition,  sanitation  and  infection  control  initiatives  in  field  hospitals. 
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Shewart  Cycle 


Agency  for  Health  Care  Policy  and  Research  1995b 


Figure  5.  Shewart  Cycle 


Her  quality  initiatives  contributed  to  a  reduction  in  the  death  rate  from  43  to  2  percent 
(Agency  for  Health  Care  Policy  and  Research  1995a).  Today’s  new  quality  measurement 
tools  measure  the  processes  of  care  provided  to  patient  populations  to  determine  if  the 
processes  achieved  good  outcomes.  These  new  criteria  emphasize  the  health  experience  of 
populations  instead  of  individual  episodes  of  care  and  focus  on  health  care  process 
management,  customer  satisfaction,  and  health  care  outcomes.  This  shift  in  focus  is  an 
attempt  to  institutionalize  total  quality  management  principles  in  health  care  delivery  and  to 
require  medical  institutions  to  adopt  a  total  quality  focus  in  their  operations. 

When  defining  health  care  quality,  most  health  care  organizations  refer  to 
Donabedian’s  health  care  quality  model  composed  of  structure,  process,  and  outcome 
(Donabedian  1980).  Structural  characteristics  consist  of  the  settings  in  which  care  takes 
place  and  have  a  propensity  to  influence  the  process  of  care  so  that  its  quality  is  diminished 
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or  enhanced.  Process  refers  to  the  interactions  which  take  place  as  medical  transactions  are 
conducted.  Changes  in  the  process  of  care,  including  variations  in  delivery,  influence  the 
effect  of  care  on  health  status,  or  outcomes  (Donabedian  1980).  Outcome  is  the  end  result 
of  the  structure/process  interaction,  and  can  be  either  immediate  and  deferred.  A  change  in 
one  aspect  of  the  structure,  process,  outcome  triad  will  necessarily  affect  the  others  because 
of  the  fundamental  functional  relationship  among  the  three  elements. 

Quality  can  also  be  defined  in  terms  of  the  interactions  between  the  provider’s 
technical  skills,  the  provider’s  interpersonal  communication  skills,  and  the  amenities  of  the 
environment  (Greenfield  and  Nelson  1992).  Technical  skills  refer  to  the  mechanics  of 
administering  care,  such  as  equipment  availability  and  efficacy  of  treatment  protocols.  The 
interpersonal  element  addresses  what  is  often  termed  the  art  of  medicine,  the  types  of 
social  interactions  between  providers  and  between  providers  and  patients.  The  amenities 
dimension  addresses  factors  at  the  institutional  level  such  as  cleanliness  of  the  facility  and 
comfort  of  the  surroundings. 

The  Typology  of  Quality  Dimensions  (Stiles  and  Mick  1994)  defines  health  care 
quality  initiatives  in  terms  of  both  sets  of  parameters  (Table  2).  This  approach  defines 
aspects  of  health  care  in  terms  of  structure,  process,  and  outcomes  and  in  terms  of 
technical,  interpersonal,  and  amenities  aspects  of  care.  The  model  classifies  quality 
initiatives  into  one  of  nine  quadrants  relating  to  the  intersection  of  the  two  quality 
definition  axes.  Classifying  initiatives  with  the  model  is  useful  in  identifying  potential 
improvement  opportunities  and  clarifying  decision  criteria.  The  model  is  also  useful  as  a 
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planning  tool  to  ensure  a  facility  addresses  all  of  the  levels  of  quality  expected  by  provider 
customers  and  other  stakeholders  (Stiles  and  Mick  1994). 


Table  2.  Typology  of  Quality  Dimensions 


Structure 

Process 

Outcome 

Technical 

Equipment 

Staffing 

Training  Programs 

Size,  Volume,  Ownership 

Accuracy  of  Diagnosis 
Appropriateness  of  Treatment 
Treatment  Skillfully  Applied 
Treatment  Plans,  Sequencing 
Practice  Guidelines 

Morbidity,  Mortality 

Increments  or  Decrements  in 
Health  or  Functional  Status 
Pallitation 

Frequency,  Distribution  of 
adverse  Incidents 

Malpractice 

Donations 

Interpersonal 

New  Technology’s  Impact  on 
Roles  and  Role  Relationships 
Building  Design,  Signage 
Presence  of  Chaplains, 
Advocates,  Social  Workers, 
etc. 

Collegiality 

Nature  of  Communication 
Honest,  Forthright 

Treatment  of  Patients 

Sensitivity  and  Compassion 

Patient  Satisfaction 

Emotional,  Spiritual  Peace 
Family  Satisfaction 

Referrals 

Compliance 

Returns  for  Future  Care 
Malpractice 

Donations 

Amenities 

Cleanliness 

Presence  of  Conveniences 

Ease  of  Access,  Parking 
Appearance  of  Staff 

Efficiency  in  Patient  Flow 

Short  Waiting  Periods 

Patient  Satisfaction 

Family  Satisfaction  j 

Referrals 

Donations 

Stiles  and  Mick  1994 


Historically  health  care  organizations  have  been  unable  to  measure  the  total  health 
of  populations  because  many  organizations  provided  only  one  aspect  of  care  in  the 
continuum  of  care.  As  the  increasing  number  of  managed  care  organizations  expand  their 
scope  of  responsibility  to  manage  health  care  settings  other  than  the  acute  care  setting, 
businesses  are  increasingly  holding  their  health  plan  accountable  not  just  for  high  quality 
treatment  of  acute  episodes,  but  for  wellness  and  prevention  of  disease  or  illness  also. 
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Definitions  of  health  are  quality  are  increasingly  shifting  away  from  uni- 
dimensional  measures  focused  on  the  presence  or  absence  of  specific  biologic  conditions 
to  more  comprehensive  measures  (Greenfield  and  Nelson  1992).  These  definitions  of 
health  and  health  care  look  beyond  physiological  and  biological  factors  to  lifestyle, 
emotional,  and  environmental  aspects  as  well.  While  traditional  definitions  of  health  and 
health  care  focus  on  the  treatment  of  disease  and  illness,  new  definitions  encompass 
concepts  such  as  patient  functional  status,  emotional  well-being  and  quality  of  life  (Figure 
7),  and  measure  quality  according  to  all  aspects  of  health  (Greenfield  and  Nelson  1992). 


Figure  7.  Health  Status  Target  Model 
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Other  innovations  include  defining  quality  of  care  in  terms  of  change  in  health 
status  as  a  result  of  intervention.  This  new  focus  on  overall  health  instead  of  episodic  care 
has  placed  a  greater  emphasis  on  preventive  medicine  and  the  quality  of  life  of  covered 
populations,  as  well  as  the  individual  patient  role  and  accountability  for  health  status. 

Peer  review  organizations  and  accrediting  organizations  are  also  using  Total 
Quality  Management  (TQM)  oriented  measures  of  quality  and  quality  improvement. 
Changes  in  the  accreditation  requirements  of  organizations  such  as  the  Joint  Commission 
for  the  Accreditation  of  Health  care  Organizations  (JCAHO)  and  the  National  Center  for 
Quality  Assurance  (NCQA),  as  well  as  the  development  of  other  quality  recognition 
processes  such  as  the  Malcolm  Baldrige  National  Quality  Award  for  Health  Care 
emphasize  the  need  for  systemic  measures  of  quality  as  opposed  to  traditional  structural 
measures  (Figure  6).  In  the  future,  health  care  systems  can  expect  customers  to  evaluate 
not  only  individual  health  care  intervention  appropriateness,  efficiency  and  effectiveness 
(efficacy),  but  the  management  of  the  health  care  processes  as  well. 

Process  Measures 

Processes  of  care  include  elements  such  as  preventive  measures,  diagnostic  tests, 
treatments,  and  other  patient  care  activities.  In  the  aggregate,  these  processes  account  for 
most  of  the  cost  of  care,  and  they  are  an  important  component  of  patient  satisfaction  or 
dissatisfaction  with  care  as  well.  Process-oriented  health  care  quality  improvement 
initiatives  focus  on  two  major  areas:  better  definition  of  the  processes  involved,  and 
evaluation  of  the  efficiency  and  efficacy  of  the  processes. 
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Health  Care  Criteria 
Framework 
Dynamic  Relationships 


Malcolm  Baldrige  1993 

Figure  6.  Malcolm  Baldrige  National  Health  Care  Quality  Award  Criteria  Framework 


Several  organizations  have  begun  compiling  information  and  creating  databases 
related  to  process  measurement.  Perhaps  the  most  well  known  database  is  the  Healthcare 
Employer  Data  Information  Set  (HEDIS)  created  by  the  National  Center  for  Quality 
Assurance  (NCQA).  HEDIS  was  established  in  1989  as  a  first  attempt  at  quantifying  for 
employers  and  consumers  those  key  quality  process  measures  attributable  to  membership 
in  managed  care  organizations  that  reflect  efficient,  effective  care  management  (National 
Committee  for  Quality  Assurance  1995a).  While  the  list  of  quality  indicators  covered  in 
the  most  recent  version  of  the  data  set,  HEDIS  2.5(Appendix  3),  is  too  large  to  discuss  in 
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detail,  the  measures  contained  in  the  data  set  generally  address  the  frequency  of  inputs  to 
the  health  process,  such  as  childhood  immunizations,  or  specific  procedures,  such  as 
mammograms  (National  Committee  for  Quality  Assurance  1995b). 

As  health  care  quality  performance  measurement  methods  become  more 
sophisticated,  employers  and  managed  care  organizations  have  begun  to  criticize  the 
HEDIS  measures  because  of  difficulties  in  data  gathering  and  interpretation  and  the  lack  of 
outcome  and  clinical  appropriateness  measures  in  the  data  set  (Harris  1994).  HEDIS 
examines  the  frequency  of  selected  processes  as  proxies  for  outcomes  and  good  practice, 
but  does  not  measure  the  efficiency,  efficacy,  or  appropriateness  of  the  processes  and 
procedures.  HEDIS  is  now  losing  favor  among  health  care  purchasers  as  a  measure  of 
managed  care  organizational  performance  because  it  is  not  results  oriented  (Bums  1995). 

Another  process-oriented  health  care  quality  measurement  endeavor  is  the  Joint 
Commission  for  the  Accreditation  of  Healthcare  Organization’s  (JCAHO’s)  Indicator 
Management  System  (IMS).  JCAHO’s  IMS  is  similar  to  HEDIS  in  many  regards,  but  has 
the  added  capability  of  continuously  monitoring  a  facility’s  processes  through  on-line 
computer  systems  (Nadzam  et  al.  1993),  while  HEDIS  measures  are  monitored  via  periodic 
reports.  Like  HEDIS,  IMS  measures  process  frequency  without  capturing  process 
outcome,  efficiency,  or  efficacy  information. 

The  IMS  has  implications  for  JCAHO  health  care  facility  accreditation  in  the  future 
as  the  system  becomes  more  wide  spread.  After  testing  and  fielding,  JCAHO  intends  to 
use  the  system  in  the  accreditation  process,  and  will  use  indicators  from  the  system  to 
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signal  follow-up  or  focus  reviews.  Facilities  not  subscribing  to  the  IMS  in  the  future  will 
not  be  accredited  by  the  JCAHO  (Nadzam  et  al.  1993). 

Outcomes  Measures 

Outcome-oriented  measures  typically  consist  of  mortality  data,  complication  rates, 
improvements  in  health  and  functional  status,  overall  quality  of  life,  and  patient  and 
provider  satisfaction  (Mitchell  1993).  Many  providers  and  purchasers  now  regard  patient 
health  outcomes  as  the  gold  standard  for  measuring  health  care  quality,  and  consider  good 
patient  outcomes  the  ultimate  goal  of  health  services  (Agency  for  Health  Care  Policy  and 
Research  1995b).  Medical  outcomes  measurement  is  a  relatively  new  endeavor,  although 
medical  outcomes  studies  have  been  conducted  since  the  days  of  Florence  Nightingale 
(Stiles  and  Mick  1994). 

Historically,  health  care  organizations  have  not  used  health  care  outcomes-oriented 
quality  measures  due  to  difficulties  with  definition  and  measurement  (Harris  1994).  To 
adequately  measure  outcomes  and  improve  the  care  delivery  process,  a  large  volume  of 
information  concerning  patients  and  the  care  delivery  process  is  needed.  Although  few 
automated  systems  exist  that  adequately  evaluate  multi-dimensional  health  status,  the 
advent  of  computers  and  the  ability  to  share  and  process  large  amounts  of  data,  have 
increased  the  development  and  use  of  outcomes-oriented  performance  measures. 

One  successful  automated  outcomes  measurement  effort  is  the  Maryland  Hospital 
Association’s  Quality  Indicator  (QI)  Project.  The  QI  Project  measures  selected  clinical 
occurrences  such  as  hospital  acquired  infections,  inpatient  mortality,  and  unscheduled 
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returns  to  the  operating  room  (Kazandjian  et  al.  1993).  These  measures,  collected  quarterly 
and  adjusted  for  severity,  are  compared  by  participating  hospital  and  identify  each 
hospital’s  rate  for  each  measure,  the  target  or  benchmark  rate  for  each  occurrence  and 
opportunities  for  improvement  (Kazandjian  et  al.  1993). 

Another  outcomes-oriented  measurement  tool  which  is  less  resource  intensive  is  the 
Short  Form  36  (SF-36)  Health  Survey  developed  by  John  Ware  and  Cathy  Sherboume 
(Appendix  4),  which  measures  multiple  aspects  of  a  patient’s  health  and  wellness.  The  SF- 
36  is  outcomes-oriented  in  that  it  measures  improvements  in  patient  functionality 
following  medical  episodes.  The  SF-36  measures  aspects  of  eight  different  health 
indicators,  and  can  be  completed  by  clinic  personnel  or  by  the  patient  (Ware  and 
Sherboume  1992).  The  SF-36  is  less  expensive  to  manage  than  an  outcomes-oriented 
database,  yet  retains  comparable  accuracy. 

As  process  and  outcomes-oriented  health  care  performance  measures  proliferate, 
health  care  provider’s  clinical  practice  variation  has  come  under  increased  scrutiny. 

Clinical  practice  variation  occurs  when  patients  with  the  same  basic  condition  receive 
different  evaluations  and  treatments.  These  practice  pattern  differences  contribute  to 
undesirable  variation  in  the  cost  of  care  and,  more  importantly,  contribute  to  variation  in 
the  quality  of  the  care  outcome. 

The  health  care  clinical  practice  variation  phenomena  is  well  documented. 
Wennburg’s  variation  study,  published  in  1973,  found  that  variations  in  the  delivery  of 
health  care  could  be  attributed  to  the  provider’s  geographic  region  of  practice,  and  that 
providers  were  treating  essentially  the  came  cases  with  different  modalities,  often  with 
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significant  differences  in  cost  and  quality  (Wennburg  and  Gittelsohn  1973).  Subsequent 
studies  have  found  variations  due  to  provider  enthusiasm  for  a  particular  procedure 
(Chassin  1993)  and  variation  influenced  by  patient  preferences  and  physiology  (Kasper, 
Mulley  and  Wennburg  1992). 

The  impact  of  practice  variation  on  both  health  care  cost  and  quality  is  illustrated 
by  the  Useful  Additions  to  Care  Diagram  (Mohlenbrock  1995)  (Figure  8).  The  diagram 
describes  care  in  term  of  two  axes,  resource  use  (cost)  and  health  benefit  (quality). 


RELATIONSHIP  BETWEEN  CLINICAL  QUALITY  AND 
INPATIENT  RESOURCE  CONSUMPTION 


I _ Resources  Consumption  (Cost) _ 

Mohlenbrock  1995 

Figure  8.  Useful  Additions  to  Care  Model 


As  resources  are  consumed  and  costs  rise,  the  health  benefit  also  rises  to  the  point  of 
maximum  cost/benefit  efficiency.  Beyond  this  point,  the  health  benefit  is  only  improved 
marginally  although  resources  are  still  consumed  at  the  same  rate,  reducing  the  utility  of 
additional  resource  inputs  and  overall  efficiency.  Eventually  continued  resource  inputs 
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will  cause  health  complications,  offsetting  the  incremental  health  benefits  of  added  inputs, 
and  will  result  in  a  decreased  health  status  for  the  patient.  The  challenge  to  health  care 
providers  is  to  provide  enough  resource  inputs  to  improve  health  status  to  the  maximum 
cost  efficiency  point,  without  over  utilizing  resources  and  negatively  influencing  cost  and 
quality  (Mohlenbrock  1995). 

One  method  to  control  variation  is  to  implement  practice  guidelines  (Agency  for 
Health  Care  Policy  and  Research  1995b).  Practice  guidelines  are  usually  developed 
through  clinical  trials  and  specify  the  most  efficient  processes  of  care  known  to  produce 
good  outcomes.  Because  they  measure  both  cost  and  quality  of  clinical  processes, 
guidelines  address  the  efficiency  and  efficacy  aspects  of  health  care  delivery. 

To  use  practice  guidelines  as  a  basis  for  quality  indicators,  the  guideline  and 
indicators  must  meet  three  conditions:  the  relationship  between  the  care  process  and  health 
outcome  must  be  known,  that  relationship  must  form  the  basis  for  the  guideline,  and  the 
quality  indicators  must  be  derived  from  the  guideline  (Agency  for  Health  Care  Policy  and 
Research  1995b).  Practice  guidelines  can  be  broad  or  narrow  in  scope  but  must  be  adapted 
to  the  local  environment  and  adhered  to  locally  if  they  are  to  be  effective.  Large  scale 
efforts  to  develop  practice  guidelines,  such  as  the  Agency  for  Health  Care  Policy  and 
Research’s  (AHCPR)  recent  practice  guideline  development  projects,  have  had  mixed 
success  in  that,  by  attempting  to  capture  all  the  elements  of  specified  health  care  delivery 
processes,  the  guidelines  become  to  broad  to  affect  cost  and  quality  locally. 

Practice  guideline  enforcement  can  be  strict  or  general  depending  on  the  specificity 
of  the  guideline  and  management  philosophy.  The  biggest  negative  associated  with  strict 
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practice  guidelines  is  that  many  providers  perceive  them  as  “cookbook”  medicine  and  a 
threat  to  their  professional  autonomy.  Consequently,  precise  practice  guidelines  are  not 
widely  accepted  among  medical  professionals  (Ellrodt  et  al.  1995). 

Implementing  practice  guidelines  to  control  variation  is  becoming  increasingly 
easier  as  computers  become  more  ingrained  in  the  health  care  environment.  Practice 
guidelines  can  be  designed  on  a  computer  and  then  the  computer  can  be  used  to  send  care 
reminders  regarding  the  most  efficient  practices  to  health  care  providers  at  critical 
junctions  in  the  care  delivery  process.  Computers  can  also  be  used  to  monitor  provider 
compliance  with  guidelines  and  to  illustrate  to  care  providers  how  their  practice  patterns 
impact  quality  and  cost  of  care  (Ellrodt  et  al.  1995). 

Often  practice  guidelines  are  developed  by  adopting  the  key  clinical  processes  of 
other  providers  to  the  current  environment,  a  process  known  as  clinical  benchmarking. 
Clinical  benchmarking  methods  compare  provider  cost  and  quality  performance  indicators 
to  others  in  a  peer  group.  While  some  managed  care  organizations  have  begun  to 
benchmark  network  provider’s  performance,  for  the  most  part  clinical  benchmarking  has 
been  conducted  by  individual  facilities  assisted  by  a  growing  industry  of  technical 
consultants  (Anderson-Miles  1995).  Because  clinical  benchmarking  is  often  an  individual 
effort,  there  is  little  standardization  among  approaches  used  by  facilities  and  consultants  as 
each  user  modifies  or  develops  benchmarking  procedures  to  match  their  environment. 

The  origins  of  benchmarking  in  industry  can  be  traced  back  to  Xerox  in  1979  when 
they  evaluated  their  delivery  processes  against  those  of  L.L.  Bean  Company  (Patrick  and 
Alba  1994).  Benchmark  best  of  practice  examples  can  be  found  within  an  organization 
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(internal  benchmarking),  from  other  organizations  within  the  industry  (external 
benchmarking),  or  from  organizations  outside  the  industry  (generic  benchmarking). 
Although  most  clinical  benchmarking  efforts  concern  clinical  practices,  many  health  care 
organizations  also  benchmark  administrative  functions  which  detract  from  quality  and  add 
cost  through  external  benchmarking  techniques.  These  organizations  benchmark  against 
best  in  class  medical  organizations  as  well  as  against  generic  functions  of  companies 
outside  the  health  care  industry  (Patrick  and  Alba  1994). 

Clinical  Benchmarking  measures  health  outcomes  and  cost  data  of  providers  or 
processes  for  specific  disease  categories  or  procedures.  These  measures  can  then  be 
compared  against  a  standard  (Bellile  1995).  Data  used  includes  mortality,  morbidity, 
length  of  stay,  charges,  DRGs  and  severity  indices  for  homogeneity.  Part  of  the  process  of 
defining  clinical  benchmarks  is  to  identify  the  profile  of  physicians  relating  to  particular 
procedures,  diagnoses  or  disease  codes.  These  profiles  must  then  be  adjusted  according  to 
patient  severity  to  allow  equal  comparison  among  providers  (Bellile  1995).  Provider 
performance  can  then  be  plotted  on  an  scatter  plot  diagram  using  quality  indicators  along 
one  axis  and  cost  indicators  along  the  other  (Figure  9).  This  method  demonstrates  central 
tendency  among  the  providers  concerning  cost  and  quality  behaviors  and  will  facilitate 
discussion  concerning  who  is  providing  the  best  care  at  the  least  cost. 
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Figure  9.  Clinical  Benchmarking  Scatter  Plot 


Clinical  benchmarking  is  gaining  acceptance  as  a  way  to  improve  the  efficiency  of 
care  delivery  while  at  the  same  time  improving  outcomes.  Clinical  benchmarking,  like 
TQM,  requires  management  support  and  provider  interest  to  be  successful.  Clinical 
benchmarking  also  reduces  an  organization’s  cost  of  providing  health  care  by  eliminating 
waste  and  rework  and  by  reducing  adverse  outcomes  related  to  unnecessary  or  duplicate 
procedures  and  tests.  Clinical  benchmarking,  when  done  correctly,  is  a  continuous  process 
of  looking  for  improvement  opportunities  and  incorporates  many  principles  of  TQM 
(Patrick  and  Alba  1994). 

While  methodological  approaches  for  developing  clinical  benchmarks  vary  widely, 
most  approaches  rely  heavily  on  automated  data  systems  to  identify  provider  resource 
usage  and  clinical  outcomes  patterns  and  then  use  statistical  programs  to  identify  best 
practices.  Most  health  care  organizations  initiating  a  clinical  benchmarking  program 
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develop  their  methodology  through  trial  and  error  as  they  work  with  their  existing  data 
system. 

Many  health  care  organizations  have  dismissed  the  feasibility  of  clinical 
benchmarking  due  to  inaccuracies  in  the  information  contained  in  their  clinical  databases. 
Although  it  is  always  desirable  to  have  perfect  data,  it  is  still  possible  to  evaluate  providers 
and  facilities  performance  using  mildly  flawed  or  inaccurate  data  through  the  technique  of 
gap  analysis  (Czamecki  1995).  Gap  analysis  uses  benchmarking  methods  to  identify 
performance  “gaps”  between  providers.  Although  the  magnitude  of  the  gap  in  these 
circumstances  is  uncertain,  often  verification  of  the  existence  of  a  gap  is  enough  to 
stimulate  process  improvement  (Czamecki  1995). 

Another  barrier  to  performing  clinical  benchmarking  is  the  lack  of  detailed 
information  concerning  health  outcomes  and  cost.  Most  managed  care  plans  do  not  capture 
detailed  outcomes  information  or  accurate  cost  data  (Hale  and  Weiner  1994).  Managed 
care  plans  do  not  have  the  information  systems  to  support  detailed  data  retrieval,  and  most 
managed  care  organizations  still  see  outcomes  management  and  quality  guidelines  as  less 
important  than  price  and  patient  satisfaction  (Hale  and  Weiner  1994).  For  those  that  desire 
to  capture  the  information  manually  by  extracting  it  from  medical  records,  the  task  is 
tedious  and  expensive,  and  the  validity  of  extracted  information  is  often  questionable  due 
to  inaccuracies  in  the  medical  records. 

To  overcome  the  data  deficit  problems  and  begin  assimilating  the  large  amounts  of 
information  required,  several  health  care  corporations  are  now  collaborating  to  share 
information  and  identify  common  best  practices  within  the  organizations.  These 
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companies  are  usually  affiliated  through  a  managed  care  system  and  are  rarely  competitors 
in  the  same  market.  A  good  example  of  such  a  corporation  is  the  SunHealth  Alliance,  a 
group  of  260  nonprofit  hospitals  located  in  15  states.  SunHealth  has  worked  to  share 
information  among  the  facilities  in  the  alliance  and  to  define  and  measure  benchmark 
processes  (Patrick  and  Alba  1994).  Additionally,  there  are  several  third  party  vendors  and 
consultants  now  offering  product  line  comparison  information  and  developing  databases 
for  comparisons.  Two  example  are  Iameter,  Inc.  and  APACHE  Systems,  both  of  which 
market  their  patented  benchmarking  products  to  health  care  networks  interested  in 
benchmarking  performance. 

Clinical  benchmarking  can  be  done  at  the  aggregate  level  such  as  hospital  or  clinic, 
or  at  the  individual  provider  level.  To  apply  clinical  benchmarking  methodology  to 
hospitals  and  other  health  care  facilities,  outcomes  and  expense  data  are  aggregated  to 
service  or  facility  level  and  are  adjusted  for  severity  according  to  the  case  mix  of  the 
facility.  Determinations  about  the  efficiency  of  processes  or  outcomes  are  affected  by  the 
individual  and  collective  characteristics  of  the  patients  who  receive  care,  the  inputs  to  care, 
such  as  demographic  and  risk  factors  and  the  patients  baseline  functional  or  health  status. 
Severity  adjusting  of  data  ensures  that  like  patients  are  being  compared  and  that  similar 
results  should  be  expected.  Without  severity  adjustment,  there  can  be  not  comparability  of 
procedures,  outcomes  or  patients  (Bellile  1995). 

Although  benchmarking  facilities  is  effective  at  controlling  cost  and  ensuring 
quality,  the  greatest  impact  made  by  benchmarking  is  at  the  provider  level,  as  the  provider 
is  the  single  largest  cost  driver  in  the  organization  (Manus,  Werner  and  Strub  1994).  By 
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comparing  individual  providers  severity  adjusted  performance,  however,  the  hospital  or 
health  care  corporation  runs  the  risk  of  potentially  embarrassing  or  alienating  some 
providers  (Ellrodt,  Conner,  Reidinger  and  Weingarten  1995).  To  preclude  this, 
comparisons  are  usually  blinded,  with  only  the  individual  provider  aware  of  their  score. 
This  convention  allows  providers  to  objectively  discuss  cost  and  outcomes  performance 
and  to  map  out  best  of  practice  processes  without  fearing  reprisals,  defensiveness,  or 
hidden  agendas.  Physicians  are  more  likely  to  accept  care  guidelines  and  clinical 
benchmarking  if  they  are  implemented  with  a  mixture  of  education,  financial  incentives 
and  disincentives,  and  appropriate  timely  feedback  concerning  their  performance  (Ellrodt, 
Conner,  Reidinger  and  Weingarten  1995). 

Clinical  benchmarking  is  consistent  with  new  process  and  outcomes  oriented 
accreditation  requirements  of  the  JCAHO  and  NCQA.  Clinical  benchmarking  also 
supports  Malcolm  Baldrige  National  Quality  Award  criteria  for  health  care  by  evaluating 
health  care  providers  based  on  patient  focus  and  value  criteria.  Because  clinical 
benchmarking  is  a  continuous  process,  it  also  supports  the  Baldrige  goals  of  continuous 
quality  improvement  and  organizational  learning.  The  clinical  benchmarking  process 
involves  the  staff  of  the  health  care  facility  in  defining  clinical  best  practices,  which  meets 
the  Baldrige  requirements  for  employee  and  health  care  staff  involvement  in  planning  care. 
Because  clinical  benchmarking  usually  utilizes  historical  medical  data  from  facility 
medical  information  systems,  it  is  fact  based  and  can  be  considered  results  oriented  in  that 
it  focuses  on  patient  outcomes  as  opposed  to  structure  or  processes. 
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Military  Medical  Information  Systems 

Civilian  health  care  providers,  especially  managed  care  providers,  are  investing 
heavily  in  health  care  information  systems  to  support  their  process  and  outcomes 
performance  measurement  efforts.  Examples  of  innovation  were  sought  from  civilian  and 
military  sources  in  an  effort  to  improve  the  methods  for  gathering  data  on  areas  such  as 
outcomes  quality,  customer  satisfaction,  and  complexity  of  processes.  Military  medical 
information  systems,  while  capable  of  supporting  limited  outcomes  oriented  quality 
performance  measures,  have  for  the  most  part  been  used  to  provide  structure  oriented 
health  care  information.  The  military  medical  information  systems  currently  in  use  do 
not  adequately  accommodate  retrieval  of  provider  level  data.  They  do  however,  support 
retrieval  of  service  and  facility  level  diagnostic,  procedure  and  cost  data,  and  can  compute 
clinical  severity  indices  as  well. 

The  MEDCOM  has  several  medical  information  systems  available  at  the 
MEDCOM  Headquarters  to  support  a  MEDCOM  Clinical  Benchmarking  effort.  The 
primary  financial  management  system  used  by  the  MEDCOM  and  the  MHSS  is  the 
Military  Expense  Performance  Reporting  System  (MEPRS).  MEPRS  data  is  driven  by 
employee  workload  reports  generated  by  the  MTFs.  These  employee  man  hour  reports 
are  gathered  in  a  centralized  database  and  converted  to  dollar  equivalents  using  accepted 
algorithms.  These  manpower  expenses  are  combined  with  other  cost  data  from  supply 
and  equipment  use  to  develop  aggregate  cost  reports.  MEPRS  can  report  cost  data  by 
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cost  center  such  as  a  clinic  or  ward,  or  by  a  clinical  service  or  department.  MEPRS 
generates  full  cost  financial  reports,  allocating  overhead  from  non-patient  care  centers  to 
patient  care  centers  according  to  Medicare  step  down  rules  (Defense  Medical  Systems 
Support  Center  1994). 

The  primary  military  medical  information  system  for  capturing  clinical  data  is  the 
PASBA2  system,  which  captures  inpatient  clinical  data  only.  PASBA2  data  is  extracted 
from  individual  inpatient  medical  records  coded  at  the  MTFs,  and  then  transmitted 
electronically  to  a  central  repository  where  it  is  severity  adjusted  and  combined  with  data 
from  other  MTFs.  PASBA2  contains  a  variety  of  clinical  indicators  and  can  be  accessed 
through  either  standard  reports  or  ad-hoc  queries  of  the  data  base  (Patient  Administration 
Systems  and  Biostatistics  Activity  1993). 

Another  clinical  information  system  available  at  the  MEDCOM  Headquarters  is 
the  Retrospective  Case  Mix  Analysis  System  (RCMAS).  RCMAS  is  different  from 
PASBA2  in  that  it  also  contains  CHAMPUS  claims  data,  to  include  outpatient 
CHAMPUS  claims  data.  Unlike  PASBA2,  RCMAS  does  not  contain  individual 
encounter  data,  but  rather  aggregates  data  for  CHAMPUS  providers,  MTFs,  and  patient 
demographic  categories.  RCMAS  is  driven  by  input  from  the  PASBA2  system  and  from 
CHAMPUS  claims  data  processed  at  the  central  CHAMPUS  claims  processing  center. 
RCMAS  is  a  flexible  system  that  supports  many  standard  reports  and  menu  driven  ad-hoc 
queries  (Defense  Medical  Systems  Support  Center  1993). 

Anticipated  future  upgrades  to  the  military  information  systems  include 
refinement  of  the  MTF  level  information  system,  the  Composite  Health  Care  System 
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(CHCS),  and  development  and  fielding  of  a  new  Computer  Executive  Information 
System  (CEIS).  CEIS  is  an  executive  level  decision  support  system  designed  to  provide 
managers  with  visibility  of  their  organization’s  operations.  CEIS  will  utilize  data  from 
the  PASB  A2  and  MEPRS  databases  and  will  not,  in  the  near  term,  supply  provider  level 
information  (CEIS  Program  Office  1995). 

Another  future  information  system  upgrade  is  the  Clinical  Information  System 
(CIS)  which  is  expected  to  be  a  computerized  patient  record.  With  the  fielding  of  the 
CIS,  a  greater  amount  of  patient  and  provider  level  data  will  be  available  through 
electronic  databases.  Detailed  information  regarding  procedures  and  processes,  in 
addition  to  provider  level  data,  will  support  accurate  MEDCOM  clinical  benchmarking 
efforts  in  the  future.  These  new  information  system  projects  will  begin  fielding  to  Army 
MTFs  in  Fiscal  Year  1997  and  will  not  be  completely  fielded  until  after  the  year  2000 

f 

(CEIS  Program  Office  1995). 


Purpose  of  the  Study 

The  purpose  of  this  study  is  to  develop  a  clinical  benchmarking  model  for 
possible  use  by  the  MEDCOM  Headquarters  staff  to  measure  the  core  business  of  the 
MEDCOM,  providing  health  care.  The  model  should  measure  health  care  value,  support 
the  strategic  goals  of  the  MEDCOM  and  MHSS,  facilitate  effective  utilization 
management,  facilitate  comparisons  with  performance  of  similar  civilian  organizations, 
be  relatively  easy  to  use  and  understand  and  make  use  of  existing  military  medical  data 
systems  to  the  maximum  extent  possible. 
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CHAPTER  2 


METHOD  AND  PROCEDURES 
Research  Plan 

The  case  study  methodology  (Yin  1989)  was  used  as  a  framework  from  which  to 
identify  and  classify  candidate  clinical  benchmarking  models  which  could  be  adapted  for 
use  by  the  MEDCOM.  Candidate  models  were  identified  through  literature  review, 
personal  interviews  with  knowledgeable  individuals  at  the  MEDCOM  and  other  facilities, 
and  by  attending  seminars  and  conferences  relating  to  clinical  benchmarking  and  health 
care  performance  measure  design.  Each  of  the  candidate  measurement  sets  and  models 
was  evaluated  to  determine  if  it  was  value  oriented,  supported  the  strategic  goals  of  the 
MEDCOM  and  MHSS,  facilitated  effective  utilization  management,  provided 
comparisons  with  civilian  organizations,  was  easy  of  use,  and  was  adaptable  to  existing 
military  medical  data  systems. 

Prior  to  evaluating  the  clinical  benchmarking  models,  available  military  medical 
information  systems  were  evaluated  to  determine  input  requirements,  processing 
characteristics  and  output  capabilities  of  the  information  systems.  In  addition  to  the 
above  literature  review  and  personal  interviews,  MEDCOM  MSCs  and  MTFs  were 
surveyed  by  mail  questionnaire  to  determine  if  they  were  currently  using  or  developing 
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measures  or  models  and  to  identify  potential  models  or  sets  already  in  use  that  could  be 
adapted  for  use  by  the  MEDCOM  Headquarters. 

Procedures 

The  research  plan  consisted  of  5  steps  intended  to  identify  existing  military 
medical  information  system  capabilities,  identify  and  classify  candidate  benchmark 
models,  evaluate  models  for  adaptability  to  the  MEDCOM  environment  and  develop  a 
MEDCOM  clinical  benchmark  model.  The  first  phase  of  the  plan  involved  a  detailed 
review  of  the  medical  and  business  literature  to  determine  how  health  care  facilities  were 
performing  process  and  outcomes  measurement  and  clinical  benchmarking.  This  phase 
also  included  personal  interviews  and  conferences  with  individuals  in  the  MEDCOM 
Headquarters  and  civilian  health  care  industry  knowledgeable  in  health  care  quality 
management.  This  phase  also  included  attendance  at  several  managed  care  and 
performance  measurement  conferences.  The  personal  interviews  and  conferences  were 
used  as  additional  resources  from  which  to  locate  applicable  information  not  available 
through  the  literature. 

Review  of  the  medical  literature  was  accomplished  using  the  MEDLINE  and 
HEALTH  databases.  Keywords  searched  in  the  databases  include:  clinical 
benchmarking,  practice  guidelines,  utilization  review  and  physician  practice  patterns. 
Results  for  the  above  searches  were  limited  to  years  1992-1996,  and  were  further 
narrowed  to  review  articles  and  abstracts.  The  years  were  limited  to  ensure  information 
regarding  only  current  practices  was  retrieved.  The  search  was  limited  to  review  articles 
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to  ensure  a  global  perspective  and  to  prevent  retrieval  of  articles  dealing  with  a  particular 
disease  or  medical  specialty.  The  search  was  further  limited  to  articles  abstracted  in  the 
database  to  assist  in  the  initial  review  process  of  selecting  appropriate  articles.  After 
initially  reviewing  the  articles  obtained  through  the  literature  search,  related  information 
concerning  practice  variations  was  retrieved  by  searching  the  HEALTH  data  base  for 
articles  by  the  author  Wennburg. 

Attendance  at  the  clinical  benchmarking  seminar  following  the  initial  literature 
review  provided  an  overview  of  current  business  practices  in  the  health  care  industry 
related  to  clinical  benchmarking.  Other  conferences  relating  to  health  care  performance 
measurement  and  transition  to  managed  care  provided  similar  results.  The  conferences 
were  an  integral  part  of  the  research  and  provided  detailed  information  regarding  current 
clinical  benchmarking  and  performance  measurement  practices  that  was  not  available 
through  the  literature.  The  conferences  also  provided  the  opportunity  to  discuss  the 
various  clinical  benchmarking  models  and  performance  measurement  sets  with  vendors 
and  users.  Input  from  MEDCOM  staff  officers  directly  involved  in  evaluating  the 
MEDCOM  strategic  plan  and  performance  measures  directed  the  research  effort  to  health 
care  performance  measures  seminars  and  numerous  medical  performance  measurement 
consultants.  Consultants  were  queried  about  their  products  and  about  particular  criteria 
related  to  operating  in  the  MEDCOM  environment. 

The  second  step  or  phase  involved  a  critical  evaluation  of  the  numerous  military 
medical  information  systems  available  to  MEDCOM  Headquarters  staff  to  identify  input 
requirements,  processing  characteristics  and  output  capabilities.  Whenever  possible, 
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interviews  with  key  personnel  responsible  for  the  operation  of  the  systems  were  used  to 
verify  system  characteristics,  and  hands-on  demonstrations  were  used  to  provide 
examples  of  input  requirements  and  output  capabilities. 

The  third  phase  consisted  of  development  of  a  survey  instrument  to  survey  the 
MEDCOM  MTFs  and  MSCs  concerning  their  use  of  performance  measures  and 
benchmarking  models  in  their  commands.  This  step  consisted  of  identifying  an  existing 
valid  survey  instrument  for  use  in  the  project,  adaptation  of  the  instrument  for  the  target 
audience,  dissemination  of  the  instrument  to  subordinate  MSCs  and  MTFs  and 
consolidation  and  analysis  of  the  returned  surveys.  Adaptation  of  an  existing  survey 
instrument  was  preferable  to  developing  an  instrument  for  this  project  to  maximize 
reliability  and  validity  of  the  instrument,  minimize  the  time  required  to  design  and 
disseminate  the  survey  and  provide  some  measure  of  comparability  of  the  results  with  the 
results  of  other  survey  efforts. 

The  instrument  chosen  to  survey  MEDCOM  organizations  for  this  project  was 
initially  developed  and  used  by  the  Government  Accounting  Office  (GAO)  in  a  1992 
project  which  surveyed  government  agencies  about  their  use  of  performance  measures 
(GAO  1992).  The  GAO  instrument  was  not  available  in  its  entirety  for  inclusion  in  this 
report.  The  instrument  was  modified  by  the  Department  of  the  Treasury  in  1993  for  use 
in  a  project  to  survey  civilian  corporations  about  their  use  of  performance  measures 
(Appendix  5)  (Project  USA  1993). 

Reliability  for  both  the  GAO  instrument  and  the  Treasury  Department  instrument 
were  reported  by  the  respective  agencies  to  be  very  good,  with  consistent  responses  from 


38 


the  surveyed  parties  of  both  studies.  Validity  of  both  instruments  has  been  determined  by 
the  respective  agencies  to  be  adequate.  The  GAO  reported  minor  discrepancies  in 
reported  information  received  via  the  instrument  and  actual  follow  up  inspection  findings, 
but  none  serious  enough  to  flaw  the  study  or  discredit  the  instrument.  The  Treasury 
Department  did  not  report  any  discrepancies  in  data  received,  nor  did  they  audit  the 
respondents  to  determine  validity.’ 

The  instrument  used  in  the  current  project  was  not  substantially  modified  from  the 
instrument  used  in  the  Department  of  the  Treasury  study,  except  to  substitute  MEDCOM 
specific  information  where  applicable  (Appendix  6).  The  validity  and  reliability  of  the 
current  project’s  instrument  is  considered  comparable  to  those  instruments.  A  potential 
problem  with  inter-rater  reliability  manifested  shortly  after  the  instrument  was 
disseminated  throughout  the  organization,  as  three  surveyed  MTFs  requested  additional 
instructions  in  completing  the  survey.  Additionally,  a  minor  item  numbering  flaw  was 
discovered  in  the  instrument  after  it  was  sent  out  to  the  respondents,  however  there  were 
no  contacts  requesting  clarification  since  the  flaw  was  discovered. 

The  instrument  was  sent  to  respondents  electronically  via  the  military  CC  mail 
system  and  facsimile.  Several  facilities  reported  that  the  electronically  transmitted 
instrument  became  distorted  and  illegible  in  transmission.  There  were  no  complaints 
concerning  the  quality  of  the  facsimile  copies.  Reliability  of  the  instrument  used  in  this 
study  was  determined  to  be  good  with  all  facilities  answering  questions  similarly. 

Validity  was  established  through  follow  up  contacts  regarding  performance  measures  and 
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benchmarking  efforts  and  reflected  that  the  questions  did  accurately  reflect  the  actions  of 
the  facilities. 

The  fourth  step  involved  evaluating  the  models  identified  in  phase  one  according 
to  the  selection  criteria  of  value  orientation,  congruence  with  the  strategic  goals  of  the 
MEDCOM  and  MHSS,  utilization  management  focus,  comparability  with  other  health 
care  organizations,  ease  of  use  and  compatibility  with  military  medical  data  systems. 

Step  four  also  involved  evaluating  the  results  of  the  MEDCOM  MTF/MSC  survey  for 
potential  models  which  met  the  above  criteria. 

To  ensure  compatibility  with  MEDCOM  information  systems,  in  the  fifth  phase 
the  proposed  MEDCOM  clinical  benchmarking  model  was  tested  with  actual  MEDCOM 
data  to  prove  the  validity  and  reliability  of  the  model  at  identifying  a  benchmark  facility. 
To  test  the  model,  individual  case  data  from  Fiscal  Year  1995  relating  to  the  MEDCOM’ s 
10  most  costly  DRGs  was  retrieved  through  the  PASBA2  system  for  each  of  the 
MEDCOM’ s  MTFs.  No  patient  identifier  data  was  retrieved  with  the  cases. 

MTF  average  cost  per  DRG  was  calculated  by  sorting  the  data  by  DRG  category 
and  MTF,  then  averaging  the  case  cost  reported  in  PASBA2  for  all  cases  of  a  particular 
DRG  at  each  MTF.  Cost  data  was  not  split  by  service  or  clinic  specialty,  but  was 
aggregated  for  the  MTF.  Lower  cost  data  indicates  better  cost  performance. 

The  quality  indicator  data  was  developed  from  secondary  diagnoses  and 
procedure  codes  reported  through  PASBA2  for  each  MTF  and  for  each  DRG  category. 
Secondary  diagnoses  and  procedure  codes  were  examined  to  identify  potential  adverse 
outcomes,  such  as  lacerations  to  internal  organs  and  infections  acquired  while  in  the 
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hospital.  These  potentially  adverse  diagnoses  were  recoded  as  “1”,  summed  for  each  case 
and  averaged  for  each  MTF.  This  number  provides  a  mean  number  of  potentially  adverse 
diagnoses  or  procedures  per  case  per  MTF.  A  higher  MTF  adverse  diagnoses  and 
procedures  score  indicates  poorer  average  quality  outcomes.  The  secondary  diagnoses 
and  procedures  selected  for  this  indicator  are  defined  in  the  International  Coding  and 
Diagnoses  manual,  version  9.  Inclusion  of  a  particular  diagnosis  or  procedure  as  an 
adverse  outcome  or  quality  indicator  should  be  determined  by  competent  medical 
authority,  preferably  by  a  panel  of  senior  specialists  in  the  field,  and  then  codified  as 


MEDCOM  policy. 


CHAPTER  3 


RESULTS 

The  first  phase  yielded  detailed  information  concerning  numerous  process  and 
outcomes  measures  and  clinical  benchmarking  methods.  Information  procured  through 
the  literature  search  is  accepted  as  complete  and  reliable.  Results  of  the  initial  search  of 
clinical  practice  guidelines  yielded  1,434  articles  in  the  database.  Results  were  narrowed 
to  1 19  articles  using  the  year  of  publication,  review  article  and  abstract  limits.  The  search 
of  utilization  review  articles  resulted  in  1,348  articles  related  to  utilization  management  in 
the  literature.  A  similar  search  of  physician  practice  patterns  resulted  in  1,948  articles  in 
the  database.  The  search  of  Wennburg  articles  resulted  in  103  articles.  Articles  were 
then  reviewed  for  applicability  to  the  research  problem  and  study  purpose,  resulting  in  5 
articles  related  directly  to  performance  measurement  or  clinical  benchmarking.  These 
articles  were  then  subjected  to  the  evaluation  criteria  to  determine  their  applicability  to 
the  MEDCOM  environment. 

The  literature  search  results  adequately  represent  the  available  literature 
concerning  health  care  oriented  performance  measures.  This  became  apparent  as  the 
information  retrieved  through  the  searches  began  to  develop  definable  characteristics  of 
the  performance  measures  which  facilitated  classification  into  the  three  elements  of  health 
care  quality:  structure,  process,  and  outcome  (Donabedian  1980).  Structure  oriented 
measures  and  articles  were  discarded  as  inappropriate  to  the  research  problem  and  study 
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purpose.  Process  and  outcomes  measures  were  evaluated  and  classified  according  to  the 
evaluation  criteria.  Validity  and  reliability  of  the  measures  and  methods 
reported  through  the  literature  were  verified  by  reviewing  each  method’s  methodological 
approach  to  the  problem  and  evaluation  of  the  repeatability  of  the  results.  Sources  were 
also  considered  adequately  valid  if  the  source  reporting  was  the  primary  source  of  the 
data,  and  if  the  source  was  reporting  actual  results  achieved  through  implementation  of 
the  methods  or  measures. 

Evaluation  of  the  existing  health  care  performance  measures  described  in  the 
literature  found  several  possible  models  that  could  be  potentially  adapted  to  the 
MEDCOM  environment.  All  models  were  reviewed  according  to  the  evaluation  criteria, 
which  were  not  weighted  and  considered  equally  important.  Models  were  evaluated  as 
mutually  exclusive,  however  elements  from  each  type  can  and  probably  should  be  used 
together  to  gain  a  more  complete  picture  of  the  organization. 

The  process  oriented  health  care  performance  measures  identified  through  the 
literature  were  the  HEDIS  2.5  and  the  JCAHO  IMS  indicator  sets.  Of  these  two,  only  the 
HEDIS  measures  were  further  evaluated,  since  the  JCAHO  indicator  set  is  not  yet 
complete  and  indicators  have  not  been  decided  (Nadzam  et  al.  1993).  HEDIS  measures 
the  number  of  health  care  processes  an  organization  performs  as  proxies  of  outcomes, 
such  as  measuring  mammograms  instead  of  number  of  late  stage  breast  cancer  patients 
(National  Committee  for  Quality  Assurance  1995).  For  this  reason,  the  HEDIS  measures 
failed  the  first  criterion  of  value  oriented.  The  HEDIS  measures  generally  support  the 
MEDCOM  and  MHSS  strategic  goals,  especially  as  they  relate  to  patient  satisfaction  and 
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market  share  issues.  The  HEDIS  measures  are  not  good  measures  of  utilization  except 
where  they  calculate  the  amount  of  administrative  overhead  and  medical  loss  ratios. 
HEDIS  measures  are  frequency  counts  of  procedures,  and  do  not  measure  the 
appropriateness  of  the  procedures  (Harris  1994).  HEDIS  measures  provide  a  great  deal  of 
comparability  with  other  health  care  organizations.  HEDIS  data  is  published  annually  by 
the  NCQA  for  nearly  50  organizations,  and  the  acceptance  of  the  measures  by  other 
health  care  organizations  is  growing  (Hale  and  Weiner  1994).  HEDIS  measures  are 
extremely  difficult  to  calculate  for  an  organization  that  is  not  capturing  the  required 
information  through  automated  systems.  Non-automated  systems  require  extensive  audit 
of  both  inpatient  and  outpatient  medical  records  to  calculate  required  ratios  and  measures 
(National  Committee  for  Quality  Assurance  1995).  For  organizations  with  automated 
systems  that  can  capture  the  required  information,  HEDIS  measures  are  very  easy  to  use, 
as  the  computer  can  be  used  to  capture  and  calculate  the  required  ratios  and  measures. 

The  measures  are  very  easy  to  use  once  they  are  calculated  and  are  easily  adapted  to 
simplistic  charts  and  graphs,  facilitating  comparisons  and  evaluation. 

The  military  medical  systems  currently  in  use  are  not  compatible  with  capturing 
and  reporting  HEDIS  measures.  The  denominator  for  most  HEDIS  measures,  the  number 
of  enrolled  beneficiaries,  is  not  defined  in  the  MEDCOM  information  systems,  and  most 
HEDIS  outpatient  quality  measures,  such  as  number  of  immunizations  and 
mammograms,  are  not  captured  by  the  military  medical  information  systems.  Some 
inpatient  information,  such  as  the  number  of  angioplasties  performed,  can  be  computed 
from  data  contained  in  the  medical  information  systems  if  the  procedure  in  question  is 
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coded  as  a  principle  procedure  or  diagnosis  for  the  patient.  Procedures  done  as  a 
intermediate  step  of  another  procedure  are  not  always  captured  unless  they  impact  the 
primary  diagnosis  of  the  patient. 

Review  of  the  literature  identified  six  indicator  sets  which  can  be  considered 
outcomes  oriented  measures.  Of  the  six  indicator  sets,  four  are  benchmarking  models. 
The  two  non-benchmarking  model’s  are  the  Clinical  Practice  Improvement  (CPI) 
methodology  and  the  SF-36  Health  Survey.  The  CPI  model  involves  applying  the 
scientific  process  to  day-to-day  patient  care  scenarios  (Horn  1995).  The  system  looks  at 
each  component  step  or  process  of  a  particular  care  delivery  process  to  determine  the  best 
content  and  timing  to  maximize  good  medical  outcomes  and  minimize  cost.  The  CPI 
model  does  not  benchmark  providers,  rather  it  examines  established  processes  of  care  to 
maximize  the  efficacy  of  the  care  processes  (Horn  1995).  The  system  relies  on  voluntary 
compliance  by  providers  as  it  does  not  provide  benchmarking  feedback  nor  provider 
profiles  to  individual  providers.  The  process  uses  patient  factors  to  severity  adjust 
processes  and  outcomes  and  provide  standardization.  In  this  regard  CPI  benchmarks 
patients,  not  providers  (Horn  1995). 

CPI  uses  the  scientific  process  to  objectively  measure  changes  in  patient  status  as 
a  result  of  interventions  (Horn  1995).  Because  the  system  measures  the  effects  of 
treatment  on  patients,  CPI  is  value  oriented.  CPI  is  congruent  with  some  of  the  strategic 
goals  of  the  MEDCOM  and  MHSS  it  that  it  facilitates  the  learning  organization 
(adaptability),  and  identifies  technology  needs,  but  does  not  provide  benchmark 
comparisons  with  other  health  care  organizations. 
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CPI  does  support  utilization  management  from  the  perspective  that  processes 
components  are  discussed  in  term  of  efficacy  and  efficiency,  however  it  was  not  clear 
how  the  tradeoff  between  the  two  was  affected,  and  there  appear  to  be  few  controls 
established  to  monitor  provider  compliance  with  established  protocols.  CPI  does  not 
provide  a  formal  comparability  mechanism  to  benchmark  performance  against  other 
facilities  and  systems,  although  best  practices  concerning  a  particular  disease  or  injury 
process  could  certainly  be  shared  among  hospitals  or  systems  (Horn  1995). 

CPI  appears  to  be  quite  complex  and  time  consuming  in  that  it  requires  collection 
of  very  detailed  patient  information  to  provide  severity  indices,  required  collection  and 
discussion  of  detailed  process  information,  not  just  for  the  best  practices,  but  for  all 
practices  relating  to  a  particular  process,  and  then  requires  collection  of  outcomes  which 
must  be  attributed  to  a  particular  intervention  or  process  change.  Results  are  not  easily 
displayed  in  graphs  and  charts,  and  require  a  great  deal  of  technical  explanation  regarding 
the  patients,  the  processes,  and  the  effects  on  outcomes. 

CPI  is  not  compatible  with  existing  MEDCOM  information  systems.  Patient 
level  information  regarding  indicators  such  as  temperature,  blood  pressure  and  lab  results 
are  not  available  at  the  aggregate  level  (Patient  Administration  Systems  and  Biostatistics 
Agencies  1993).  Additionally  processes  in  the  MTFs  vary  widely  from  facility  to  facility 
and  are  not  captured  in  any  one  area  for  review.  Outcomes  measures  such  as  patient 
functional  status  and  satisfaction  are  not  currently  available,  nor  are  long  term  medical 
outcomes  since  our  systems  only  capture  inpatient  clinical  information. 
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The  SF-36  Health  survey  developed  by  John  Ware  and  Cathy  Sherboume  is  a 
multi-item  questionnaire  that  measures  eight  different  health  domains  relating  to  health 
and  functionality  (Ware  and  Sherboume  1992).  Responses  to  the  survey  questions  are 
provided  by  the  patient  and  can  be  obtained  through  the  mail,  by  telephone,  or  in  person. 
The  survey  can  be  administered  to  patients  shortly  after  and  acute  treatment  episode  or 
several  months  later  to  determine  the  impact  of  treatment  on  health  status.  Because  the 
survey  measures  health  status  and  functionality  following  interventions  it  is  considered 
value  oriented.  Because  the  instrument  is  relatively  new,  there  is  not  enough  information 
available  to  determine  if  it  provides  the  required  level  of  detail  to  effectively  manage 
outcomes. 

The  SF-36  meets  some  of  the  strategic  goals  of  the  MEDCOM  and  MHSS  in  that 
it  is  health  and  fitness  focused  and,  when  combined  with  utilization  data,  provides 
information  on  the  effectiveness  of  treatment.  The  SF-36  does  not  measure  patient 
satisfaction  (Ware  and  Sherboume  1993).  The  SF-36  is  not  a  particularly  good  utilization 
management  tool,  in  that  it  does  not  provide  information  regarding  the  efficiency  of  the 
care  delivery  process.  When  combined  with  the  treatment  specific  information,  it  does 
provide  a  good  indication  of  the  efficacy  of  the  treatment. 

The  SF-36  makes  no  provisions  for  benchmarking  performance  or  for  profiling 
providers.  In  the  broadest  sense,  the  SF-36  provides  some  comparability  to  other  health 
facilities  and  systems,  should  they  adopt  the  form.  Because  the  questions  are  answered 
subjectively  by  patients  rather  than  objectively  by  researchers,  the  reliability  and 
credibility  of  the  data  is  questionable.  The  survey  alone  does  not  provide  a  clears  enough 
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picture  of  the  patient  population  or  care  process  characteristics.  Output  is  clear  and 
concise  and  is  easily  trended  with  graphs  and  charts. 

The  largest  benefit  of  the  tool  is  that  it  is  very  easy  to  use  and  adapts  well  to 
automated  data  input  devices  such  as  scanners  and  light  pens.  Because  the  form  captures 
data  first  hand  from  patients  there  is  a  requirement  to  initially  set  up  a  data  base  and 
information  management  hardware  system  to  support  compilation  of  the  data.  This  is  the 
same  for  all  organizations  however,  since  the  form  does  not  draw  from  automated  data 
systems. 

The  other  four  outcomes  oriented  measurement  sets  are  all  varieties  of  clinical 
benchmarking  models.  The  models  are  the  APACHE  medical  cost  management  system, 
the  Maryland  Hospital  Association’s  QI  Project,  The  SunHealth  Alliance  clinical 
benchmarking  model  and  the  Iameter  clinical  benchmarking  model.  All  of  these  systems 
work  similarly  to  compare  severity  adjusted  facility  cost  and  quality  data  across  facilities 
or  providers.  The  focus  and  processes  associated  with  each  model  are  slightly  different 
from  model  to  model,  as  are  the  input  requirements.  The  APACHE  model  is  cost  focused 
rather  than  quality  focused,  and  benchmarks  facility’s  severity  adjusted  inpatient  cost 
center  performance  (APACHE  Medical  Systems  Inc.  1995).  The  Maryland  QI  Project 
model  is  value  oriented,  and  benchmarks  facility’s  severity  adjusted  medical  outcomes 
indicators  for  selected  clinical  occurrences  (Kazandjian  et  al.  1993).  The  SunHealth 
Alliance  model,  like  the  APACHE  model,  focuses  on  cost  rather  than  outcomes  as  it 
benchmarks  processes  of  care  as  opposed  to  outcomes  of  care.  SunHealth  data  is  not 
severity  adjusted  as  it  does  not  measure  the  outcomes  of  the  processes,  only  the  cost  and 
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the  components.of  the  process  (Patrick  and  Alba  1994).  The  Iameter  benchmarking 
model  incorporates  both  clinical  outcomes  data  and  cost  data  to  provide  a  total  value 
profile  for  a  facility.  Iameter  has  several  methods  of  severity  adjusting  their  data  based 
on  the  need  of  a  particular  client  or  facility  (Mohlenbrock  1995). 

All  of  the  benchmarking  models  are  consistent  with  MEDCOM  and  MHSS 
strategic  goals  in  that  they  benchmark  to  find  best  of  class  performance  and  then  change 
the  organization’s  behavior  to  adopt  those  processes  and  procedures.  All  models,  through 
the  review  of  practice  guidelines  and  treatment  protocols,  can  assist  the  organization  in 
identifying  new  technology  and  processes  to  assist  in  the  adaptation  to  the  environment. 
There  were  no  clear  cut  differences  between  the  models  in  this  regard,  except  the  Iameter 
model  which  benchmarks  both  cost  and  quality  data  to  provide  a  complete  picture  of  the 
organizations  performance. 

The  systems  were  different  in  their  approaches  to  utilization  management.  The 
APACHE  system  and  SunHealth  system  monitor  the  efficiency  of  the  system  through 
cost  analysis,  while  the  QI  Project  model  measured  the  effectiveness  or  efficacy  of 
treatment  but  not  the  efficiency.  Only  the  Iameter  model  captured  data  on  the  clinical 
appropriateness  of  processes,  the  efficiency  of  the  organization,  and  the  efface  of  the 
treatments. 

All  of  the  systems  provide  excellent  comparability.  All  models  are  in  wide  use 
throughout  the  United  States,  are  employed  in  a  variety  of  facilities  and  measure  many 
key  processes  and  procedures.  Because  of  differences  in  types  of  data  gathered,  focus  of 
analysis,  and  differences  in  severity  adjustment  processes,  the  models  do  not  facilitate 
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comparison  with  other  models  data.  Each  of  the  models  has  a  large  central  data 
repository  and  is  capable  of  generating  periodic  reports. 

Reports  generated  by  the  models  are  all  relatively  easy  to  use  and  adapt  well  to 
graphical  presentations.  The  models  provide  clear  data  that  can  be  trended  internally  or 
benchmark  against  other  facilities.  The  data  from  the  Iameter  model  can  be  more 
complex  since  it  calculates  length  of  stay  and  cost  together  on  a  scatter  plot  diagram  by 
deviations  from  the  mean  rather  than  by  absolute  numbers.  This  results  in  a  diagram 
where  the  best  demonstrated  performance  is  in  the  upper  right  of  the  graph  as  opposed  to 
a  graph  of  actual  numbers,  which  would  place  the  benchmark  in  the  lower  left  comer. 

All  of  the  systems  have  some  degree  of  compatibility  with  military  medical 
information  systems.  The  APACHE  system  uses  length  of  stay  and  inpatient  cost  center 
level  cost  data  to  measure  performance  (APACHE  Medical  Systems  Inc.  1995),  both  of 
which  are  available  through  MEDCOM  information  systems.  The  QI  Project  model  uses 
a  variety  of  inpatient  and  outpatient  indicators  to  measure  clinical  performance 
(Kazandjian  et  al.  1993).  Ambulatory  measures  are  not  captured  in  existing  data  systems 
although  inpatient  indicators,  such  as  mortality,  infections,  and  unscheduled  readmissions 
are  captured  (Patient  Administration  Systems  and  Biostatistical  Activities  1993).  The 
SunHealth  model  requires  cost  data  by  hospital  department  for  a  particular  Diagnosis 
Related  Group  (DRG)  or  procedure  (Patrick  and  Alba  1994).  Most  of  the  data  required  is 
available  through  existing  MEDCOM  information  systems.  Aggregate  information 
regarding  average  cost  per  DRG  is  available  through  both  PASBA2  and  MEPRS.  The 
Iameter  model  measures  outcomes  with  mortality  data  and  length  of  stay,  both  of  which 
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are  available  through  the  military  medical  information  systems.  The  Iameter  model 
measures  cost  performance  by  average  charges  or  cost  per  Medical  Diagnostic  Category 
and  DRG  (Mohlenbrock  1995),  information  which  is  available  through  the  military 
medical  information  systems. 

Attendance  at  the  two  seminars  provided  first  hand  accounts  of  health  care 
personnel  responsible  for  implementing  and  using  the  systems  at  their  facilities.  In 
general,  results  obtained  from  the  measures,  models,  and  systems  were  positive.  The 
most  obvious  result  from  the  conferences  was  the  visible  increased  interest  in  health  care 
performance  measurement.  This  was  evident  from  the  number  of  speakers  representing  a 
variety  of  organizations,  as  well  as  from  first  hand  accounts  of  experiences  from  the 
attendees.  Attendance  at  the  seminars  illustrated  the  widespread  acceptance  of  the  need 
to  benchmark  in  health  care  and  the  variety  of  performance  measures  available  to 
facilities. 

Results  from  the  MEDCOM  survey  illustrated  the  need  within  the  MEDCOM  for 
an  effective  benchmarking  tool.  Although  many  MTFs  responded  that  they  had  clinical 
benchmarking  measures  in  place,  few  facilities  could  quantify  them.  Most  performance 
measures  used  by  the  MTFs  are  outdated  MEDCOM  structure-oriented  measures  which 
do  not  reflect  process  and  outcomes.  Several  MTFs  reported  having  clinical  practice 
guidelines  published  but  response  was  not  constant  across  the  MEDCOM  and  compliance 
with  the  guidelines  is  generally  not  required  or  measured.  AMEDD  practice  guidelines 
appear  to  be  only  loosely  coordinated  within  MTFs  and  appear  to  be  more  suggestions  for 
care  rather  than  absolute  criteria. 
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One  facility  was  noteworthy  in  that  it  captured  and  trended  performance  of 
emergency  room  process  improvements,  benchmarking  internally.  These  measures  were 
not  exportable  to  the  MEDCOM  Headquarters  however,  since  the  measures  were 
developed  with  information  systems  not  available  at  the  headquarters.  Several 
respondents  conveyed  frustration  with  the  current  MEDCOM  management  indicators, 
reinforcing  the  headquarters  staff  s  conclusions  that  the  measures  were  out  of  date. 

Others  indicated  that  while  they  gathered  and  analyzed  a  large  amount  of  data,  most  of 
their  efforts  had  little  impact  on  the  organizations  effectiveness. 

Results  from  the  review  of  the  military  medical  information  systems  indicate  that 
information  reliability  and  validity  varies  from  poor  to  very  good.  Information  from  the 
PASBA2  system  in  most  instances  has  been  proven  reliable  and  valid  by  extensive  use 
and  testing  and  is  monitored  through  rigorous  quality  checks  prior  to  inclusion  in  the  data 
base  (Patient  Administration  Systems  and  Biostatistical  Agencies  1993).  Reliability  and 
validity  of  the  MEPRS  data  was  not  perceived  by  users  as  good  due  to  the  high  potential 
for  input  errors  by  MTF  personnel.  MEPRS  is  reflective  of  manpower  allocations  to 
work  centers,  and  currently  there  is  poor  enforcement  of  accurate  reporting  across  the 
MEDCOM  (Defense  Medical  Systems  Support  Center  1994).  This  was  verified  during 
the  course  of  interviewing  individuals  associated  with  quality  measurement  in  the 
military  medical  system.  Several  individuals  related  anecdotal  information  concerning 
input  errors  which  raised  questions  about  the  reliability  and  validity  of  the  data. 
Reliability  and  validity  of  RCMAS  data,  which  comes  from  PASBA2  for  the  MTF 
portion  and  from  CHAMPUS  claims  for  the  CHAMPUS  portion,  was  determined  to  be 
very  good.  PASBA2  quality  procedures  have  already  been  reported,  and  CHAMPUS 
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claims  data  is  drawn  from  the  actual  claims  filed  by  the  beneficiaries  and  providers 
(Defense  Medical  Systems  Support  Center  1993).  CHAMPUS  data  is  not  reflective  of 
the  entire  universe  of  civilian  care  provided  to  military  beneficiaries,  but  only  for  those 
episodes  for  which  a  beneficiary  files  a  claim  and  a  payment  is  made.  Therefore,  care 
must  be  taken  in  comparing  the  data  from  the  two  sources  as  the  CHAMPUS  data  should 
be  considered  incomplete. 
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CHAPTER  4 
DISCUSSION 

Results  indicate  several  viable  alternatives  meeting  the  MEDCOM  MTF  clinical 
performance  measure  criteria.  Comparison  of  the  performance  measures  identified 
through  the  literature  search  indicate  that  some  models  are  better  suited  for  MEDCOM 
application  than  others  although  all  models  met  the  selection  criteria  to  some  extent. 
Results  are  summarized  in  Table  3. 


Table  3.  Performance  Measure  Model  Comparison 


Value 

Oriented 

Meets 

Strategic 

Goals 

Supports 

UM 

Compar¬ 

ability 

Ease  of 
Calculation 

Compatibili 
ty  with 
MEDCOM 
Info. 

Systems 

HEDIS 

Measures 

no 

yes 

yes 

yes 

yes 

no 

CPI  Model 

yes 

yes 

yes 

no 

no 

no 

yes 

yes 

no 

no 

yes 

no 

APACHE 

no 

yes 

yes 

yes 

yes 

yes 

QI  Project 

yes 

yes 

yes 

yes 

yes 

yes 

SunHealth 

no 

yes 

yes 

yes 

yes 

yes 

Iameter 

yes 

yes 

yes 

yes 

yes 

yes 
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Of  the  models  presented  in  the  literature,  the  Iameter  and  QI  Project  models  seem  to  best 
meet  the  criteria  established  for  evaluation.  They  provide  value  data,  are  congruent  with 
MEDCOM  and  MHSS  strategic  goals,  provides  good  comparability  to  other  health  care 
organizations,  are  reasonably  easy  to  use  and  prepare,  and  are  compatible  with  the 
existing  military  medical  information  systems. 

The  CPI  model,  QI  Project  model  and  Iameter  benchmarking  model  all  fulfill  the 
first  criteria  of  measuring  health  care  value,  as  does  the  SF-36  survey.  The  HEDIS 
measures,  APACHE  model  and  SunHealth  benchmarking  model  do  not  measure  value. 
The  HEDIS  indicator  set  measures  measure  the  activity  of  processes  which  are  presumed 
to  be  indicators  of  outcomes,  but  doesn’t  measure  the  result  or  outcome  of  the  process. 
The  APACHE  and  SunHealth  models  are  cost  focused  and  do  not  measure  the  outcomes 
of  the  clinical  procedures.  The  SF-36  survey  is  the  only  instrument  or  model  that  looks  at 
long  term  outcomes  in  terms  of  functionality.  The  CPI  model,  Iameter  benchmarking 
model  and  the  QI  Project  model  all  examine  mortality  and  complications  as  outcomes, 
which  are  relatively  short  term  outcomes  related  to  episodic  delivery  of  care.  While 
appropriate  in  the  acute  care  setting,  these  models  do  not  accurately  measure  the  impact 
of  the  continuum  of  care  on  a  patient  population’s  health  status  or  wellness. 

All  of  the  measures  discussed  in  the  literature  support  measurement  of  progress 
toward  MEDCOM  and  MHSS  strategic  goals  to  some  extent.  The  HEDIS  measures 
provide  measures  of  patient  satisfaction,  but  do  little  to  facilitate  the  learning  organization 
or  to  evaluate  managed  care  processes.  Although  process  measures  are  currently  in  use 
by  many  managed  health  care  organizations,  the  literature  indicates  that  acceptability  and 
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enthusiasm  for  the  measures  is  waning  and  their  ability  to  accurately  reflect  an 
organizations  performance  is  questionable  (Bums  1995).  CPI  supports  the  learning 
organization  and  assists  in  the  identification  of  appropriate  uses  for  new  technology,  but 
does  not  measure  patient  or  provider  satisfaction,  disease  prevention  or  market  share  and 
penetration. 

The  APACHE,  SunHealth  and  Iameter  benchmarking  models  support  strategic 
goals  of  a  learning  organization  and  identification  and  application  of  new  technology. 
They  also  support  sharing  of  best  clinical  practices  based  on  quality  and  cost  control 
within  the  MEDCOM  and  with  other  health  care  organizations.  The  SF-36  survey 
instrument  supports  the  strategic  plan  by  measuring  the  effectiveness  of  the  entire 
military  health  care  system  at  managing  the  populations  health  and  wellness,  an 
overarching  principle  of  the  strategic  plan.  The  SF-36,  like  the  HEDIS  measures  do  little 
to  support  the  learning  organization  and  technology  related  goals,  and  do  not  provide 
good  utilization  management  data.  Both  models  require  additional  analysis  to  decide  if 
the  services  used  are  in  fact  appropriate. 

The  HEDIS  model  identifies  the  frequency  of  use  of  modalities  of  care  perceived 
to  be  effective,  but  is  not  flexible  enough  to  recognize  more  efficient  modalities  as  they 
are  identified  or  to  identify  appropriateness  of  the  modalities  when  they  are  used.  The 
SF-36  identifies  the  end  product  of  the  health  care  process,  but  does  not  measure  the 
efficiency  of  the  process.  The  CPI  model  could  potentially  be  useful  for  UM,  but  is  not 
clear  in  its  presentation  of  efficiency  versus  effectiveness.  The  QI  Project  identifies  key 
outcomes  to  evaluate  the  effectiveness  of  treatment,  but  does  not  provide  feedback 
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concerning  the  efficiency  of  the  delivery  process.  The  other  benchmarking  models  all 
provide  excellent  cost  effectiveness  and  clinical  efficacy  in  formation,  however  only  the 
Iameter  model  has  a  clinical  effectiveness  component  to  evaluate  the  health  care  value. 

All  of  the  models  with  the  exception  of  the  SF-36  and  CPI  model  are  currently 
being  used  by  the  civilian  health  care  sector  and  offer  a  variety  of  benchmarking  partners 
and  comparisons.  Although  HEDI’S  measures  are  the  most  well  known,  the  literature 
indicates  that  they  are  losing  favor  as  more  organizations  move  to  outcomes  oriented 
models  and  clinical  benchmarking  models.  All  of  the  benchmarking  models  provide 
excellent  comparability  to  other  facilities  and  providers  in  the  industry,  and  most  are 
national  in  scope  with  several  facilities  throughout  the  United  States.  The  SF-36  and  CPI 
are,  by  design,  individual  facility  or  system  oriented  measures  which  make  comparisons 
among  facilities  difficult.  The  SF-36  could  potentially  be  benchmarked  against  other 
facilities,  but  there  are  currently  no  established  databases  of  this  information,  and 
comparison  of  SF-36  data  would  require  developing  a  case  mix  adjustment  methodology. 
The  CPI  model  does  not  provide  performance  comparisons,  although  it  does  facilitate 
sharing  of  best  processes.  Measurement  of  the  cost  and  quality  results  of  the  processes 
are  not  a  part  of  the  model,  although  they  could  be  captured  and  shared  as  well. 

Output  from  the  CPI  model  is  by  far  the  hardest  to  prepare  and  understand. 
Because  the  model  focuses  on  defining  the  best  processes  of  care,  extensive  discussion 
related  to  all  possible  care  modalities  is  required  to  effectively  identify  the  best  processes. 
Implementation  of  these  processes  requires  another  extensive  training  period,  and 
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measurement  of  the  effectiveness  of  the  revised  procedures  is  dependent  upon  the 
complexity  of  the  process  and  the  number  of  intermediate  steps  altered. 

HEDIS  measures  and  SF-36  survey  data  are  the  easiest  to  compute  and  understand 
since  they  consist  of  simple  ratios  and  are  easily  displayed  in  graphs  and  charts. 

Likewise,  the  benchmarking  model’s  data  is  also  easy  to  compute  and  lends  itself  well  to 
graphic  display,  however  these  measures  are  slightly  more  difficult  to  explain  as  they 
involve  variations  in  care  processes  and,  in  some  cases,  discussion  of  outcomes  measures. 
With  the  exception  of  the  SF-36  data,  outcomes  measures  are  more  difficult  to  compute 
and  require  a  fair  amount  of  initial  education  to  use  effectively.  HEDIS  measures  are 
vulnerable  to  wrong  interpretations  and  miscalculations,  especially  if  the  measures  are 
calculated  by  hand.  Benchmarking  models  are  sensitive  to  the  severity  adjustment 
criteria,  and  care  must  be  take  to  ensure  like  items  are  compared. 

HEDIS,  CPI  and  SF-36  models  all  require  information  that  is  not  currently 
available  through  MEDCOM  information  systems.  The  benchmarking  models  all  use 
information  that  is  available  to  some  degree  through  existing  information  systems.  The 
QI  Project,  to  be  fully  compatible,  requires  ambulatory  data  which  is  not  available.  The 
SunHealth  model  requires  department  level  cost  data  which  is  available  as  clinic  work 
center  cost  data. 
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CHAPTER  5 
RECOMMENDATIONS 

The  MEDCOM  should  explore  development  of  a  clinical  benchmarking  model 
similar  to  the  Iameter  benchmarking  model.  While  Iameter  would  certainly  agree  to 
establish  and  maintain  a  measurement  system  for  the  MEDCOM,  it  is  possible  to  develop  a 
comparable  system  internally. 

Another  recommendation  is  to  combine  aspects  of  several  of  the  outcomes  models  to 
look  at  outcomes  data  relevant  to  the  MEDCOM  environment.  Such  a  model  could 
incorporate  quality  elements  from  the  QI  Project  model  and  compare  these  with  cost  data 
from  the  Iameter  model  to  establish  internal  benchmarks  for  the  MEDCOM  MTFs.  Utility 
of  this  model  would  be  expanded  if  the  MEDCOM  could  identify  facilities  or  systems  from 
the  civilian  sector  using  a  similar  method  not  discussed  in  the  literature  from  which  to  draw 
external  benchmarking  data.  If  such  a  partner  could  not  be  located,  then  it  is  possible  to 
identify  external  benchmarks  for  each  axis,  quality  and  cost,  independently. 

Using  the  existing  data  bases  to  design  a  MEDCOM  specific  benchmarking  model  to 
evaluate  cost  and  quality  has  other  benefits  as  well.  The  large  clinical  and  financial  data 
bases  make  the  MEDCOM  an  attractive  benchmarking  partner  for  other  military  and  civilian 
health  care  organizations.  The  databases  also  support  internal  benchmarking  efforts  since 
most  MEDCOM  MTFs  perform  similar  procedures  and  contribute  data  to  the  data  bases. 
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In  implementing  a  clinical  benchmarking  model,  the  first  step  should  involve 
identification  of  the  MEDCOM’ s  ten  most  costly  DRGs  to  identify  where  the  greatest 
potential  savings  exist.  Benchmarking  by  DRG  category  negates  the  requirement  to  severity 
adjust  the  data  since  all  diagnoses  and  procedures  within  a  particular  DRG  category  are 
considered  equally  resource  intensive.  These  DRGs  should  then  be  evaluated  by  the 
specialty  consultant  responsible  for  that  DRG  to  identify  quality  indicators.  Potential  quality 
indicators  available  through  existing  systems  are  mortality,  complication,  or  re-admission 
data. 

After  identifying  the  quality  criteria,  MTF  cost  and  quality  performance  for  each 
DRG  can  be  plotted  on  a  scatter-plot  graph  to  identify  high  and  low  performers.  The 
benchmark  facility  is  the  facility  with  the  lowest  average  cost  per  procedure  and  the  best 
average  quality  per  procedure.  Thresholds  for  quality  indicators  should  be  decided  by  the 
clinical  specialty  consultant  responsible  for  that  particular  DRG  prior  to  benchmarking  to 
prevent  benchmarking  a  facility  with  unacceptable  quality.  Following  identification  of  best 
performers,  each  “best  of  class”  facility  can  document  their  procedures  and  processes  for 
other  MEDCOM  facilities  to  allow  them  to  adopt  the  “most  efficient”  practices. 
Implementation  of  the  best  practices  at  other  MTFs  is  easily  verified  by  periodically 
reviewing  updated  benchmarking  cost  and  quality  data.  Facilities  that  fail  to  improve  or  that 
improve  beyond  the  benchmark  performance  process  can  discuss  their  efforts  with  others  in 
the  command  and  possibly  redefine  the  best  practice  procedures. 

Figure  10  illustrates  the  application  of  the  proposed  MEDCOM  benchmarking 
model.  After  examining  all  MEDCOM  inpatient  workload  for  Fiscal  Year  1995,  DRG  373, 
Vaginal  Delivery  Without  Complicating  Diagnoses,  was  identified  as  the  most  costly  DRG 
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performed  by  the  MEDCOM  in  Fiscal  Year  1995.  Using  information  available  through 
MEPRS  and  PASBA2,  each  MTF’s  average  cost  and  quality  performance  during  Fiscal  Year 
1995  for  DRG  373  was  plotted  on  a  graph.  To  reduce  the  potential  for  anomalies  and 
outliers,  MTFs  performing  fewer  than  35  DRG  373  procedures  were  not  plotted.  Cases  were 
grouped  by  MTF,  not  by  clinical  specialty,  so  that  points  on  the  graph  for  MTFs  with  more 
than  one  clinical  service  performing.DRG  373  reflect  the  aggregate  performance  of  the 
MTF,  not  the  performance  of  a  particular  specialty  or  clinic. 


MEDCOM  BENCHMARKING  MODEL 


DRG  373  -  VAGINAL  DELIVERY  W/OUT  CC 


Figure  10.  MEDCOM  Benchmarking  Model 


To  protect  the  identities  of  the  facilities,  each  facility  was  assigned  a  sequence 
number.  Only  the  facility  commander  or  their  representative  should  know  their  facility’s 
sequence  number  to  ensure  anonymity.  Averages  for  each  axis,  cost  and  quality,  were  not 
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calculated  in  the  model.  While  the  averages  may  be  useful  for  comparing  facilities,  this  is 

T 

not  the  intended  purpose  of  the  model.  Rather,  the  model  is  intended  to  identify  the  MTF 
providing  the  best  health  care  value,  which  would  be  the  MTF  with  the  lowest  average  cost 
and  fewest  average  adverse  diagnoses.  MTF  performance  relative  to  an  average  is 
irrelevant. 

Three  MTFs,  numbers  three,  eight,  and  1 1  appear  to  be  potential  benchmarks.  MTF 
number  three  has  the  best  outcomes  on  average,  but  is  more  expensive  than  the  other  two.  In 
an  environment  seeking  to  maximize  quality  at  any  expense,  this  would  be  the  benchmark 
facility.  MTF  number  1 1  has  the  lowest  average  cost,  but  has  a  higher  number  of  adverse 
outcomes,  on  average,  than  the  other  two.  In  an  environment  seeking  to  minimize  cost 
without  regard  to  quality,  this  could  be  a  benchmark  facility.  MTF  number  eight  appears  to 
offer  a  compromise  between  cost  minimization  and  quality  maximization.  While  MTF 
eight’s  average  costs  are  higher  than  MTF  11  ’s,  average  quality  is  better.  Conversely,  when 
compared  to  MTF  three,  MTF  eight  has  poorer  outcomes  on  average,  but  offers  some  cost 
savings  on  average. 

To  decide  which  MTF  is  the  best  benchmark,  appropriate  MEDCOM  medical 
personnel  need  to  establish  minimum  quality  thresholds.  After  the  thresholds  have  been 
established,  the  benchmark  MTF’s  commander  and  service  chief  can  then  be  asked  to 
document  the  processes  of  care  which  led  to  the  benchmark  performance,  and  share  them 
with  the  other  MTF  Commanders  and  service  chiefs.  After  a  reasonable  amount  of  time, 
DRG  cost  and  quality  indicators  for  each  of  the  MTFs  should  be  re-measured  to  determine  if 
the  other  MTFs’  have  been  successful  at  implementing  the  new  processes,  and  to  determine 
if  a  new  benchmark  has  been  established. 
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CHAPTER  6 


CONCLUSIONS 

Changes  in  the  health  care  and  military  environments  require  the  MEDCOM  to 
better  manage  health  care  utilization  and  quality  in  MEDCOM  MTFs.  Effective 
utilization  management  is  possible  by  benchmarking  key  clinical  procedures  and 
processes,  identifying  facilities  which  are  high  performers,  and  adopting  best  practices  at 
other  facilities.  By  exploring  the  clinical  benchmarking  methods  used  by  civilian  health 
care  organizations,  a  similar  model  can  be  developed  for  use  in  MEDCOM  MTFs. 

The  clinical  benchmarking  methods  developed  by  this  project  are  useful  to 
MEDCOM  decision  makers  attempting  to  ensure  the  MHSS  remains  the  health  plan  of 
choice  for  military  beneficiaries.  Use  of  clinical  benchmarking  measures  allows  timely 
identification  of  deficiencies  in  policy  and  practice  and  facilitates  aggressive  responses  to 
potential  environmental  threats. 

The  clinical  benchmarking  model  developed  by  this  project  assists  with  UM  by 
identifying  facilities  and  procedures  incurring  unnecessary  costs,  and  by  providing  an 
efficient  benchmark  for  comparison.  Health  care  cost  can  be  decreased  as  duplicate 
procedures,  waste  and  rework  are  eliminated  and  access  will  be  improved  as  providers 
increase  efficiency  by  reducing  the  number  of  unnecessary  procedures  and  resources  are 
redirected  to  providing  services  for  other  patients.  Additionally,  clinical  benchmarking 
can  improve  health  care  quality  as  providers  adopt  the  clinical  practices  resulting  in  good 
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outcomes.  The  greatest  benefit  from  this  project  will  come  from  the  ability  to  objectively 
demonstrate,  through  cost  and  quality  outcomes  measures,  that  the  MHSS  provides  better 
health  care  value  than  civilian  competitors,  is  the  best  managed  care  system,  and  the  most 
logical  health  care  choice  for  beneficiaries. 
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APPENDIX  1. 


CURRENT  MEDCOM  HEALTH  CARE  PERFORMANCE  MEASURES 


RESOURCES 

Funding  Per  Beneficiary 

Human  Resources 

Officer  Assigned  Percentage 
Enlisted  Personnel  Assigned 
Civilian  Workyears 

Infrastructure 

Number  of  Hospitals  over  40  Years  Old 

Number  of  Clinics  Over  40  Years  Old 

Average  Number  of  Exam  Rooms  Per  Primary  Care  Provider 

PERFORMANCE  OUTCOMES 

Cost 

Cost  Per  RWP 

Cost  of  CHAMPUS/MCS  Contracts  Per  Capita 
Cost  Per  Beneficiary 

Utilization  Management 

Bed  Days  Per  1000  Beneficiaries  (D/AD) 

Average  Length  of  Stay 
Inpatient  Case  Mix  Index 
Average  Length  of  Stay/Case  Mix  Index 
Clinic  Visits  Per  Capita  (D/AD) 

Average  Daily  Inpatient  Load 
Admissions  Per  1,000  Beneficiaries 

AREAS  OF  SPECIAL  INTEREST 
Catchment  Population 
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APPENDIX  2. 


PERFORMANCE  MEASURES 
DEFINITION  REFERENCE  SHEET 


1.  Resource  Inputs  -  Recurring  resources  provided  for  your  organization's  product  or 
service  activities  (e.g.,  dollars,  staff,  materials,  etc.). 

2.  Work/Activity  Level  -  Work  measures  that  assess  intermediate  steps  in  producing 
outputs  (e.g.,  number  of  applications  in  process,  usage  rates,  inventory  levels,  etc.). 

3.  Complexity  of  Work  Processes  -  Measures  of  level(s)  of  difficulty  associated  with 
work  processes  or  activities  (e.g.,  product  refinement  vs.  new  product  development). 

4.  Internal  Measures  of  Quality  of  Products  or  Services  -  Measures  of  quality  from 
internal  sources  (e.g.,  quantitative  scores  by  quality  reviewers,  error  rates,  etc.). 

5.  External  Customer  Needs  -  Measures  of  product  or  service  expectations  and 
enhancements  by  users. 

6.  External  Customer  Satisfaction  -  Measures  of  quality  and  timeliness  from  external 
sources  (e.g.,  external  customer  surveys,  levels  of  complaints,  etc.). 

7.  Timeliness  of  Products/Services  -  Measures  of  intervals  required  to  complete  a  task 
or  measures  based  on  past  trends. 

8.  Outputs  or  Final  Products  -  Products  or  services  produced,  distributed,  or  provided 
to  service  population  (e.g.,  number  of  customers  served,  number  of  forms  processed, 
quantity  of  goods  produced,  etc.). 

9.  Financial  Measures  -  Comparison  of  planned  vs.  actual  expenditures,  costs, 
obligations,  receipts,  allocations,  or  losses. 

10.  Efficiency  Measures  -  Measures  such  as  cost  per  unit,  productivity  measures,  ratios 
of  direct  to  indirect  costs,  etc. 
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APPENDIX  3 


HEDIS  2.5  PERFORMANCE  MEASURES 


Quality/Access 

Childhood  Immunizations 

Cholesterol  Screenings 

Mammography 

Pap  Smear 

Prenatal  Care  Visit 

Diabetic  Retinal  Exam 

Major  Affective  Disorder  Follow-up 

Members  Visiting  Provider 

Athsma  Admission  Rate 

Low  Birthweight  Rate 

Physician  Network 

Physician  Turnover 
Board  Certification 

Utilization 

Coronary  Bypass  Rate 

Angioplasty  Rate 

Cardiac  Catheterization  Rate 

Cholecystectomy  Rate 

Hysterectomy  Rate 

Prostatectomy  Rate 

Laminectomy  Rate 

Cesaerian  Section  Rate 

Obstetrical  Hospital  Stay 

Readmission  for  Chemical  Dependency 

Hospital  Days/1000 

Membership/Finance 

Member  Disenrollment 

Medical  Loss  Ratio 

Administrative  Loss  Ratio 

Revenue  Requirements  Per  Member  Per  Month 
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APPENDIX  4 


SHORT  FORM  36  SURVEY 
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APPENDIX  4  . 


WARE  AND  SHERBOURNE 


Medical  Care 


Appendix.  SF-36  Questions" 

1.  In  general,  would  you  say  your  health  is: 

2.  Compared  to  one  year  ago.  how  would  you  rate  your  health  in  general 
now? 

3  The  following  items  are  about  activities  yoa  might  do  during  a  typical 
day.  Does  your  health  now  limit  you  in  these  activities?  If  so.  how  much? 

a.  Vigorous  activities,  such  as  running,  lifting  heavy  objects,  pamcipat* 
ing  m  strenuous  sports 

b.  Moderate  activities,  such  as  moving  a  table,  pushing  a  vacuum 
cleaner,  bowling,  or  playing  golf 

c.  Lifting  or  carrying  groceries 

d.  Climbing  several  flights  of  stairs 

e.  Climbing  one  flight  of  stairs 

f.  Bending,  kneeling,  or  stooping 

g.  Walking  more  than  a  mile 

h.  Walking  several  blocks 

i.  Walking  one  block 

j.  Bathing  or  dressing  yourself 

4.  During  the  past  4  weeks,  have  you  had  any  of  the  following  problems 
with  your  work  or  other  regular  daily  activities  as  a  result  of  your  physi¬ 
cal  health? 

a.  Cut  down  the  amount  of  time  you  spent  on  work  or  other  activities. 

b.  Accomplished  less  than  you  would  like 

c.  Were  limited  in  the  kind  of  work  or  other  acnvities 

d.  Had  difficulty  performing  the  work  or  other  activities  (for  example, 
it  took  extra  effort) 

5.  During  the  past  4  weeks,  have  you  had  any  of  the  following  problems 
with  your  work  or  other  regular  daily  acnvities  as  a  result  of  any  emo¬ 
tional  problems  (such  as  feeling  depressed  or  anxious)? 

a.  Cut  down  the  amount  of  time  you  spent  on  work  or  other  activities 

b.  Accomplished  less  than  you  would  like 

c.  Didn't  do  work  or  other  activities  as  carefully  as  usual 

6.  During  the  past  4  weeks,  to  what  extent  has  your  physical  health  or 
emotional  problems  interfered  with  your  normal  social  activities  with 
family,  friends,  neighbors,  or  groups? 

7.  How  much  bodily  pain  have  you  had  during  the  past  4  weeks? 

8.  During  the  past  4  weeks,  how  much  did  pain  interfere  with  your  normal 
work  (including  both  work  outside  the  home  and  housework)? 

9.  These  questions  are  about  how  you  feel  and  how  things  have  been 
with  you  during  the  past  4  weeks.  For  each  question,  please  give  the  one 
answer  that  comes  closest  to  the  way  you  have  been  feeling.  How 
much  of  the  time  during  the  past  4  weeks 
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a.  Did  you  feel  full  of  pep? 

b.  Have  you  been  a  very  nervous  person? 

c.  Have  you  feit  so  down  in  the  dumps  that  nothing  could  cheer  you 
up? 

d.  Have  you  feit  calm  and  peaceful? 

e.  Did  you  have  a  lot  or  energy? 

f.  Have  you  feit  downhearted  and  blue? 

g.  Did  you  feel  worn  out? 

h.  Have  you  been  a  happy  person? 

i.  Did  you  feel  nred? 

10.  During  the  past  4  weeks,  how  much  of  the  time  has  your  physical  health 
or  emotional  problems  interfered  with  your  social  activities  (like  visiting 
with  friends,  relatives,  etc.)? 

11.  How  TRUE  or  FALSE  is  each  of  the  following  statements  for  you? 

a.  1  seem  to  get  sick  a  little  easier  than  other  people 

b.  1  am  as  healthy  as  anybody  I  know 

c.  I  expea  my  health  to  get  worse 

d.  My  health  is  excellent 


SF*36  Response  Choices4 

1.  Excellent.  Very  Good.  Good.  Fair.  Poor 

2.  Much  better  now  than  one  year  ago.  Somewhat  better  now  than  one 
year  ago.  About  the  same  as  one  year  ago.  Somewhat  worse  now  than 
one  year  ago.  Much  worse  than  one  year  ago 

3.  Yes.  Limited  a  lot:  Yes.  Limited  a  little;  No.  Not  limited  at  ail 

4a-d.  Yes.  No 

5a-c.  Yes.  No 

6.  Not  at  all.  Slightly.  Moderately,  Quite  a  bit.  Extremely 

7  None.  Very  mild.  Mild.  Moderate.  Severe.  Very  severe 

8  Not  at  all.  A  little  bit.  Moderately.  Quite  a  bit.  Extremely 

9  All  of  the  time.  Most  of  the  time.  A  good  bit  of  the  time.  Some  of  the 
time.  A  little  of  the  time.  None  of  the  time 

10.  All  of  the  time.  Most  of  the  time.  Some  of  the  time.  A  little  of  the  time. 
None  of  the  time 

11.  Definitely  true.  Mostly  true.  Don't  know.  Mostly  false.  Definitely  false 

'Copyright •’  The  MOS  Trust.  Inc.;  1990.  All  rights  reserved. 
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Survey  of  Private  Sector  Council 
Performance  Measures 


INTRODUCTION 

The  Private  Sector  Council  is  collaborating  with  the 
Financial  Management  Service’s  Project  USA  staff  to 
develop  models  of  excellent  management  practices  for 
the  financial  improvement  of  the  Federal  government. 
Government  leaders  have  recognized  the  need  for 
Federal  agencies  to  measure  program  performance  and 
to  use  these  measures  to  improve  program  management 
and  to  annually  report  on  progress  toward  agency  goals 
and  objectives. 

We  have  developed  this  survey  to  identify  key 
performance  measurement  practices  used  in  the  private 
sector.  As  a  Private  Sector  Council  member,  your 
company’s  experience  and  expertise  would  provide  us 
with  a  valuable  illustration  of  effective  performance 
measurement  that  could  be  applied  to  the  Government. 

In  this  survey  we  are  attempting  to  obtain  the  following 
information:  1)  a  description  of  performance  measures 
used  in  your  organization,  2)  an  understanding  of  how 
performance  measures  are  used  in  conjunction  with  your 
organization’s  planning  efforts,  and  3)  an  indication  of 
how  satisfied  you  are  with  your  performance  measures. 


INSTRUCTIONS 

Most  of  the  questions  in  this  questionnaire  can  be 
answered  by  checking  boxes  or  making  short  written 
entries.  Please  be  assured  that  all  replies  will  be  kept 
confidential.  Throughout  the  survey,  the  term  "your 
organization"  refers  to  all  divisions  of  your  corporation, 
not  just  the  finance  function. 

Please  return  this  survey  by  April  3,  1992  to  Karen 
Pedone  at  the  Financial  Management  Service  by  mail 
or  by  FAX.  The  address  and  FAX  information  are 
located  on  the  last  page  of  the  survey. 

If  you  have  any  questions  about  the  survey,  please 
contact  Karen  Pedone  at  (202)  874-7074. 


We  THANK  YOU  for  taking  time  from  your  busy 
schedule  to  complete  this  survey  and  we  sincerely 
appreciate  your  continued  commitment  to  our 
Government’s  effectiveness. 


Performance  Measures 

•Performance  measures"  are::  a  composite  of 
!«y  indicators  of:  a:  pm 
mputs; ;  outputs,'  'outcomes;  pnwiuiativity, 
timeliness;:  and/or  "quality.  They  are :  means 
of.  evaluating  products  or  femces  by 
measuring  :  the  results:  against  agreed  upon 
goals  or  standards. 

If  your  organization.:  does:  not  have  product 
or:  service  ddivery;  ^ 

.substitute- 

performance..  measures*  fbr:  "product:  .or 
service  performancemeasures.*^i#^:l 


Name  of  corporation: 


RESPONDENT  INFORMATION 


Name: 


Title: 


Address: 


Telephone  Number: 

C _ ) 


B  - 1 


PRIVATE  SECTOR  COUNCIL 
SURVEY  OF  PERFORMANCE  MEASURES 
(continued) 


L  TYPES  OF  MEASURES 

1.  For  the  following  types  of  performance  measures,  please  indicate  whether  the  measure  is  currently  in  active  use  in 
your  organization,  whether  it  is  under  development  in  your  organization,  or  whether  it  is  neither  in  use  nor  under 
development  in  your  organization. 


: 

Performance 

Measure 

Currently 

In  Use 

Performance 

Measure 

Under 

Development 

Neither 

In  Use 

Nor  Under 
Development 

Performance  Measures  and  Definitions 

Resource  Inputs  -  Recurring  resources 
provided  for  your  organization’s  product 
or  service  activities  (e.g.  dollars, 
staff,  materials,  etc.) 

39 

0 

2 

Work/ Activity  Level  -  Work  measures  that 
assess  intermediate  steps  in.  producing  outputs 
(e.g.  number  of  applications  in  process, 
usage  rates,  inventory  levels,  etc.) 

35 

4 

3 

Complexity  of  Work  Processes  -  Measures 
of  Ievel(s)  of  difficulty  associated  with 
work  processes  or  activities  (e.g.  product 
refinement  vs.  new  product  development.) 

16 

4 

22 

Internal  Measures  of  Quality  of  Products 
or  Services  -  Measures  of  quality  from 
internal  sources  (e.e.  Quantitative  scores 
by  quality  reviewers,  error  rates,  etc.) 

37 

6 

2 

External  Customer  Needs  -  Measures  of 
product  or  service  expectations  and 
enhancements  by  users. 

37 

5 

1 

External  Customer  Satisfaction  -  Measures 
of  Quality  and  timeliness  from  external 
sources  (e.g.  external  customer  surveys, 
levels  of  complaints,  etc.) 

37 

6 

1 

Timeliness  of  Product  or  Services  -  Measures 
of  intervals  required  to  complete  a  task  or 
measures  based  on  past  trends. 

31 

3 

7 

Outputs  or  Final  Products  -  Products  or 
services  produced,  distributed,  or  provided 
to  service  population  (e.g.  number  of  customers 
served,  number  of  forms  processed,  quantity  of 
goods  produced,  etc.) 

38 

2 

1 

Financial  Indicators  -  Comparison  of  planned 
vs.  actual  expenditures,  costs,  obligations, 
receipts,  allocations,  or  losses. 

41 

1 

0 

Efficiency  Measures  *  Measures  such  as  cost 
per  unit,  productivity  measures,  ratios 
of  direct  to  indirect  costs,  etc. 

40 

2 

0 
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PRIVATE  SECTOR  COUNCIL 
SURVEY  OF  PERFORMANCE  MEASURES 
(continued) 


2.  Who  are  the  current  users  of  the  performance  measures  that  currently  exist  in  your  organization?  (CHECK  ALL 
BOXES  THAT  APPLY.) 

Note:  Line  units  are  those  involved  in  the  production  of  a  product  or  service  for  an  customer  e.g.  Sales, 

Engineering,  Research  and  Development,  and  Purchasing.  Staff  units  provide  internal  corporate  support  for  line’ 
functions  e.g.  Accounting,  Human  Resources,  or  Information  Resources. 


Types  of  measures 
that  exist  in 
your  organization: 

Reid/ 

Plant 

Line 

Unit 

Reid/ 

Opera¬ 

tions 

Staff 

Unit 

"  Head¬ 
quarters 
Staff 
Organ¬ 
ization 

Chief 

finan¬ 

cial 

Officer 

(CFO) 

Chief 

Execu¬ 

tive 

Officer 

(CEO) 

Customers 

Stock¬ 

holders 

Resource  Inputs 

36 

34 

33 

29 

24 

4 

7 

Work/Activity  Level 

38 

34 

28 

19 

10 

2 

0 

Complexity  of  Work 
Processes 

19 

16 

14 

2 

1 

0 

0 

Internal  Measures 
of  Quality  of 

Products  or  Services 

37 

35 

26 

12 

14 

4 

1 

External  Customer 

Needs 

33 

33 

31 

13 

18 

10 

2 

External  Customer 
Satisfaction 

32 

32 

32 

16 

21 

12 

2 

Timeliness  of 

Product  or  Service 

34 

31 

23 

12 

10 

7 

1 

Outputs  or  Final 

Products 

39 

37 

31 

18 

19 

4 

4 

Financial  Indicators 

40 

37 

39 

41 

40 

9 

23 

Efficiency  Measures 

39 

37 

37 

34 

34 

6 

10 

3 
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SURVEY  OF  PERFORMANCE  MEASURES 
(continued) 


II.  MEASUREMENT  DEVELOPMENT 


3.  Does  your  organization  currently  have  performance  measures  under  development? 


A.  27  Yes  ->  (CONTINUE  WITH  QUESTION  4.) 

B.  14  No  — >  (SKIP  TO  QUESTION  5.) 


4.  For  those  performance  measures  currently  under  development  in  your  organization,  please  indicate  the  reasons 
why  they  are  being  developed. 


Performance 

Measures: 

Unit- 

Initiated 

Improvement 

Efforts 

Company 

Mandated 

Customer 

Driven 

Other 

(Please  Specify) 

Resource  Inputs 

7 

8 

4 

1 

Work/Activity  Level 

12 

5 

2 

2 

Complexity  of  Work 

Processes 

8 

2 

2 

1 

Internal  Measures 
of  Quality 

16 

7 

8 

3 

External  Customer 

Needs 

14 

6 

10 

3 

External  Measures  of 

Customer  Satisfaction 

16 

9 

9 

2 

Timeliness  of 

Product  or  Service 

12 

7 

7 

1 

Outputs  or  Final 

Products 

9 

7 

5 

1 

Financial  Indicators 

9 

8 

3 

3 

Efficiency  Measures 

12 

12 

3 

1 

B-4 
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PRIVATE  SECTOR  COUNCIL 
SURVEY  OF  PERFORMANCE  MEASURES 
(continued) 


5.  In  your  organization,  which  performance  measures  are  generally  developed  by  the  following  people 
or  processes?  (CHECK  ALL  BOXES  THAT  APPLY.) 


Performance 

Measures: 

By  staff 
managers 

By  line 
managers 

By  employee 
focus  groups 

By  customer/ 
client  input 

Other  (Please 
specify) 

Resource  Inputs 

29 

33 

8 

2 

1 

Work/ Activity  Level 

25 

33 

10 

1 

2 

Complexity  of  Work 

Processes 

16 

20 

7 

1 

0 

Internal  Measure's  of 

Quality 

33 

37 

21 

8 

2 

External  Customer  Needs 

27 

28 

18 

30 

3 

External  Measures  of 
Customer  Satisfaction 

27 

27 

18 

31 

2 

Timeliness  of  Product 
or  Service 

25 

31 

15 

17 

2 

Output  or  Final  Product 

28 

36 

13 

8 

2 

Financial  Indicators 

38 

26 

5 

2 

5 

Efficiency  Measures 

37 

37 

12 

3 

5 

TTT  USE  OF  PERFORMANCE  MEASURES  IN  PRODUCT/SERVICE  AND  ORGANIZATIONAL  PLANNING 


6.  Please  indicate  whether  your  organization  prepares  written  plan(s)  that  set  long-term  product  or 
organizational  goals,  standards,  or  objectives.  (CHECK  ALL  THAT  APPLY.) 


A.  3  A  single  long-term  plan  that  sets  goals,  standards,  or  objectives  for  the  entire  organization  or 

product. 

B.  14  Individual  long-term  plans  that  set  goals,  standards  or  objectives  for  the  organization  or  product 

are  prepared  by  the  organization’s  sub-units. 

C.  24  No  written  plan(s)  are  prepared  that  set  long-term  product  or  organizational  goals,  standards,  or 

objectives. 
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PRIVATE  SECTOR  COUNCIL 
SURVEY  OF  PERFORMANCE  MEASURES 


7.  Considering  all  of  the  goals,  standards,  or  objectives 
contained  in  your  plan(s),  bow  many  are  written  in  terms 
that  are  measurable  or  quantifiable?  (CHECK  ONE.) 

A.  6  All 

B.  27  Most 

C.  5  About  half 

D.  3  Some 

E.  I _ ]  Few  or  none 


10.  Does  your  organization  require  the  following  to  be  refl 
in  senior  management  performance  appraisals?  (CHI 
ALL  THAT  APPLY.) 

A.  39  Goals,  standards,  and  objectives  in  your 

organization’s  plan(s) 

B.  38  Performance  Measures 

C.  21  Total  Quality  objectives 


8.  Considering  all  of  the  goals,  standards,  or  objectives 
contained  in  your  long-term  plan(s),  how  many  are 
translated  into  annual  or  semi-annual  operating  plans? 
(CHECK  ONE.) 

A.  19  All 

B.  119  Most 

C.  1  About  half 

D.  2  Some 

E.  I _ ]  Few  or  none 


11.  Does  your  organization  relate  the  following  to  s 
management  incentive  plans?  (CHECK  ALL  T 
APPLY.) 


A.  39  Opals,  standards,  and  objectives  in  your 

organization's  plan(s) 

B.  38  Performance  measures 

C.  21  Total  Quality  objectives 


9.  How  many  of  your  organization’s  goals,  standards,  or 
objectives  included  in  your  pian(s)  have  related  measures 
of  progress?  (CHECK  ONE.) 

A.  4  All 

B.  24  Most 

C.  8  About  half 

D.  3  Some 

E.  ( _ ]  Few  or  none 


12.  Is  there  a  central  office  in  your  organization  th 
responsible  for  managing  or  monitoring  the  coilectic 
internal  distribution  of  all  or  most  p»rfm  nnw.  mea 
for  the  entire  organization?  (CHECK  ONE) 

A.  28  No 

B.  13  Yes  — >  Please  provide  the  following: 

Name  of  Central  Office: 

Person  responsible: 

Title: _ 

Phone  Numbertf  t 
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PRIVATE  SECTOR  COUNCIL 
SURVEY  OF  PERFORMANCE  MEASURES 
(confirmed) 


IV.  REPORTING  AND  USE  OF  MEASURES 


13.  Are  current  performance  measures  disttibuted  internally 
as  a  single  report  to  top  organization  management? 
(CHECK  ONE.) 


A.  8  Yes  —  >  How  many  times  per  year  is  the 

report  issued? 

11(1)  12(6)  ?(1) _ Times  per  year 

B.  28  No,  there  are  multiple  reports 

C.  5  No,  there  are  no  reports 


14.  What  key  performance  measurement  indicators  and  ratios 
does  your  corporation  use?  e.g.  number  of  errors  per 
volume,  projects  completed  per  time  elapsed,  net 
assets/sales,  etc.  (If  possible,  please  attach  copies  of  your 
organization’s  indicators  and  ratios.  Confidentiality  is 
guaranteed.) 

(See  Appendix  C) 


15.  Does  your  corporation  benchmark?  (Benchmarking  is 
defined  as  searching  for  the  best  practices  that  will 
help  define  superior  products,  services,  or  support 
processes.) 


A.  34  Yes  —  >  (CONTINUE  WITH  QUESTION  16) 

B.  7  No  —  >  (SKIP  TO  QUESTION  18) 


16.  Does  your  corporation  benchmark  the  following? 


17.  To  what  does  your  organization  compare  its  performan 
measures?  (CHECK  ALL  THAT  APPLY.) 

A.  32  To  historical  trends  within  the  same  product 

service  line. 

B.  33  To  product  or  service  goals,  standards, 

objectives  (e.g.  in  annual  operating  or  strate: 
plans.) 

C.  26  To  other  units  or  regions  within  the  organiz: 

D.  32  To  alternative  suppliers  or  similar  operatic 

(e.g.,  other  companies  or  competitors.) 

E.  23  To  the  best  domestic  organizations. 

F.  27  To  world  class  organizations. 

G.  3  Other  (Please  specify), 

.  * 

H.  0  No  comparisons  are 

18.  Does  your  corporation  use  automated  measuremi 
information  systems? 

A.  29  Yes  — >  If  so,  is  the  system: 

3  An  off  the  shelf  variety? 

22  Custom  designed? 

4  No  answer 

B.  12  No 


A.  3  Indicators  and  Ratios 

B.  1  Methods  or  Processes 

C.  30  Both 
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PRIVATE  SECTOR  COUNCIL 
SURVEY  OF  PERFORMANCE  MEASURES 
(continued) 


V.  SATISFACTION  WITH  USE  OF  PERFORMANCE  MEASURES  IN  MANAGEMENT  PROCESS 


How  satisfied  or  dissatisfied  are  you  with  your  current  performance  measures  as  a  tool  to  hein  vour 
organization  do  the  following?  F  3 


Very 

satisfied 

Somewhat 

satisfied 

Neither 
satisfied  nor 
dissatisfied 

Somewhat 

dissatisfied 

Very 

dissatisfied 

Make  budget 
decisions 

7 

21 

9 

3 

1 

Manage  products 
or  services 

5 

21 

7 

8 

0 

Assure  accountability 

10 

16 

8 

7 

0 

Measure  product/service 
results  or  outcomes 

7 

22 

7 

5 

0 

20.  Overall,  how  satisfied  or  dissatisfied  is  your  organization  with  the  rinwitnac^  reliability, 

the  information  produced  by  your  organization’s  current  performance  measures?  ' 

and  level  of  detail 

Very 

satisfied 

Somewhat 

satisfied 

Neither 
satisfied  nor 
dissatisfied 

Somewhat 

dissatisfied 

Very 

dissatisfied 

Timeliness  of  the 
information  produced 
by  the  measures 

8 

21 

6 

5 

1 

Reliability  of  the 
information  produced 
by  the  measures 

15 

14 

10 

2 

0 

Level  of  detail  of 
the  information 
produced  by  the 
measures 

8 

16 

10 

7 

0 
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PRIVATE  SECTOR  COUNCIL 
SURVEY  OF  PERFORMANCE  MEASURES 
(continued) 


VL  APPLICABILITY  OF  PERFORMANCE  MEASUREMENT  TO  GOVERNMENT 


21.  Do  you  think  the  federal  government  should  implement  a  performance  system? 

A.  39  Yes  —  >  (CONTINUE  WITH  QUESTION  22.) 

B.  2  No  —  >  (SKIP  TO  QUESTION  23.) 


22.  Please  indicate  the  types  of  performance  measures  that  you  would  recommend  be  used  for  the  following 
purposes: 


Performance 

Measures: 

Measure 

Products/ 

Services 

Manage 

Products/ 

Services 

Assure 

Account* 

ability 

Make 

Budget 

Decisions 

Resource  Inputs 

25 

28 

30 

33 

Work/Activity 

Level 

23 

30 

25 

22 

Complexity  of  Work 

Processes 

14 

21 

10 

10 

Internal  Measures 
of  Quality 

29 

30 

28 

10  ** 

External  Customer 

Needs 

29 

30 

21 

22 

External  Measures  of 

Customer  Satisfaction 

29 

28 

29 

19 

Timeliness  of 

Product  or  Service 

28 

32 

29 

14 

Outputs  or  Final 

Products 

26 

31 

27 

24 

Financial  Indicators 

19 

31 

33 

36 

Efficiency  Measures 

24 

33 

31 

34 

9 


B-9 


PRIVATE  SECTOR  COUNCIL 
SURVEY  OF  PERFORMANCE  MEASURES 
(continued) 


23.  Do  you  have  any  other  recommendations  for  implementing  performance  measurement  in  the  Federal 
Government? 

(See  Appendix  D) 


24.  Please  provide  any  additional  comments  you  may  have. 


(See  Appendix  E) 


PLEASE  RETURN  THIS  SURVEY  BY  AFREL3f1992TOi 
Mailing  Address: 

FinanciaI.ManagementS5iivicei.»  g  ||f|f 
•  Roomr313 '.Obeity -Center: . .  -  . 
40Ti4tkst;..$av*::  i: 
washingto^D:c;:2022^gg::o. 

Fax  numbers:  (202)2874)735. 

(202)  8744743? 

Thank  you  VERY  MU CH  for: yourtimeandassistance. 
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MEDCOM  PERFORMANCE  MEASURES  SURVEY 
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APPENDIX  6 

MEDCOM  PERFORMANCE  MEASURES  SURVEY 


INTRODUCTION 

The  U.S.  Army  Medical  Command,  (MEDCOM),  in  an  effort  to 
provide  meaningful  information  to  commanders  at  all  levels,  is 
currently  exploring  options  to  re-engineer  the  current  MEDCOM 
Performance  Measures.  We  have  developed  this  survey  to  identify 
key  performance  measurement  practices  used  in  MEDCOM  activities. 
As  a  part  of  the  MEDCOM,  your  organization's  experience  and 
expertise  will  provide  us  with  a  valuable  illustration  of  useful, 
effective  performance  measurement  that  can  be  applied  throughout 
the  MEDCOM. 

In  this  survey  we  are  attempting  to  obtain  the  following 
information:  1)  a  description  of  performance  measures  used  in 
your  organization,  2)  an  understanding  of  how  performance 
measures  are  used  in  conjunction  with  your  organization's 
planning  efforts,  and  3)  an  indication  of  how  satisfied  you  are 
with  your  performance  measures . 


INSTRUCTIONS 

Most  of  the  questions  in  this  questionnaire  can  be  answered 
by  checking  boxes  or  making  short  written  entries.  When 
completing  the  survey,  we  ask  that  you  refer  to  activities  of  all 
divisions  within  your  organization,  not  just  the  Resource 
Management  function.  As  you  complete  the  survey,  we  ask  that  you 
take  time  to  especially  consider  new  and  innovative  performance 
measures  used  in  your  organization,  especially  those  related  to 
process  and  outcomes  evaluation.  This  instrument  is  not  intended 
to  measure  compliance  with  existing  MEDCOM  guidance,  but  rather 
to  share  good  ideas  and  best  practices. 

Please  return  this  survey  by  December  8,  1995  to  CPT  Gregory 
Swanson  at  the  Plans,  Analysis,  and  Evaluation  Directorate  of  the 
MEDCOM  by  mail  or  FAX.  The  address  and  FAX  information  are 
located  on  the  last  page  of  the  survey.  If  you  have  any 
questions  about  the  survey,  please  contact  CPT  Swanson  at 
Commercial  (210)  221-7142  or  DSN  471-7142. 

We  THANK  YOU  for  taking  time  from  your  busy  schedule  to 
complete  this  survey  and  we  sincerely  appreciate  your  continued 
commitment  to  the  MEDCOM' s  effectiveness. 


Name  of  Activity: 
RESPONDENT  INFORMATION 
Name : 

Title: 

Address : 


Telephone  Number: 
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I.  TYPES  OF  MEASURES 

1.  For  the  following  types  of  performance  measures,  please 
indicate  whether  the  measure  is  currently  in  active  use,  under 
development,  or  neither  in  use  nor  under  development  in  your 
organization . 


Performance 
Measure 
Currently 
In  Use 


Performance 

Measure 

Under 

Development 


Neither 
In  Use 
Nor  Under 
Development 


Performance  Measures  and  Definitions 


Resource  Inputs  -  Recurring  resources 
provided  for  your  organization's  product 
or  service  activities  (e.g.  dollars, 
staff ,  materials ,  etc . ) 

Work/Activity  Level  -  Work  measures  that 
assess  intermediate  steps  in  producing 
outputs (e.g.  number  of  applications  in 
process,  usage  rates,  etc.) 

Complexity  of  Work  Processes  -  Measures 
of  level (s)  of  difficulty  associated  with 
work  processes  or  activities  (e.g.  new  patient 
vs.  follow  up  visit.) 

Internal  Measures  of  Quality  of 
Products  or  Services  -  Measures  of 
quality  from  internal  sources 
(e.g.  quantitative  scores  by  quality 
reviewers,  error  rates,  etc.) 

External  Customer  Needs  -  Measures  of 
product  or  service  expectations  and 
enhancements  by  users. 

External  Customer  Satisfaction  -  Measures 
of  quality  and  timeliness  from  external 
sources  (e.g.  external  customer  surveys, 
levels  of  complaints,  etc.) 

Timeliness  of  Product  or  Services  - 

Measures  of  intervals  required  to  complete 
a  task  or  measures  based  on  past  trends. 

Outputs  or  Final  Products  -  Products  or 
services  produced,  distributed,  or 
provided  to  service  population  (e.g. 
number  of  customers  served,  number  of 
forms  processed,  quantity  of  goods 
produced,  etc.) 
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Performance 
Measure 
Currently 
In  Use 


Performance 

Measure 

Under 

Development 


Neither 
In  Use 
Nor  Under 
Development 


Financial  Indicators  -  Comparison  of 
planned  vs.  actual  expenditures,  costs, 
obligations ,  receipts ,  allocations , 
or  losses. 

Efficiency  Measures  -  Measures  such  as 
cost  per  unit,  productivity  measures,  ratios 
of  direct  to  indirect  costs,  etc. 


2.  Who  are  the  current  users  of  the  performance  measures  that 
currently  exist  in  your  organization?  (CHECK  BOXES  THAT  APPLY.) 
Note:  Line  units  are  those  involved  in  the  production  of  a  product 
or  service  for  an  external  customer  e.g.  clinics.  Staff  units 
provide  internal  support  for  Line  functions  e.g.  Human  Resources  or 
Logistics. 


Operating 

Agency 

Line 

Unit 


Operating 

Agency 

Staff 

Unit 


MEDCOM 

Head¬ 

quarters 

Staff 


Activity 

Commander 


Customers 


Types  of  measures: 

Resource  Inputs 

Work/Activity  Level 

Complexity  of  Work 
processes 

Internal  Measures 
of  Quality  of 
Products  or  Services 

External  Customer 
Needs 

External  Customer 
Satisfaction 

Timeliness  of 
Product  or  Service 

Outputs  or  Final 
Products 

Financial  Indicators 
Efficiency  Measures 
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II .  MEASUREMENT  DEVELOPMENT 


3 .  Does  your  organization  currently  have  performance  measures  under 
development? 

A.  Yes  --->  (CONTINUE  WITH  QUESTION  4.) 

B.  No  --->  (SKIP  TO  QUESTION  5.) 


4 .  For  those  performance  measures  currently  under  development  in 
your  organization,  please  indicate  the  reasons  why  they  are  being 
developed. 


Unit-  MEDCOM 

Initiated  Mandated 

Improvement 
Efforts 


Customer 

Driven 


Other 

(Please 

Specify 

Below) 


Performance 
Measures : 


Resource  Inputs 

Work /Activity  Level 

Complexity  of  Work 
Processes 

Internal  Measures  of 
Quality 

External  Customer 
Needs 

External  Measures  of 
Customer  Satisfaction 

Timeliness  of 
Product  or  Service 

Outputs  or  Final 
Products 

Financial  Indicators 
Efficiency  Measures 
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5.  In  your  organization,  which  performance  measures  are  generally 
developed  by  the  following  people  or  processes?  (CHECK  ALL  BOXES 
THAT  APPLY.) 


By  staff 
managers 


Performance  Measures: 


By  line  By  employee  By  customer/  Other 
managers  focus  groups  client  input  (Please 

(Specify) 


Resource  Inputs 

Work /Activity  Level 

Complexity  of  Work 
Processes 

Internal  Measures  of 
Quality 

External  Customer  Needs 

External  Measures  of 
Customer  Satisfaction 

Timeliness  of  Product 
or  Service 

Output  or  Final  Product 
Financial  Indicators 
Efficiency  Measures 


III.  USE  OF  PERFORMANCE  MEASURES  IN  PRODUCT/SERVICE  AND 
ORGANIZATIONAL  PLANNING 


6.  Please  indicate  whether  your  organization  prepares  written 
plan(s)  that  set  long-term  (greater  than  one  year)  product  or 
organizational  goals,  standards,  or  objectives.  (CHECK  ALL  THAT 
APPLY.) 

A.  A  single  long-term  plan  that  sets  goals,  standards,  or  objectives 
for  the  entire  organization  or  product. 

B.  Individual  long-term  plans  that  set  goals,  standards  or 
objectives  for  the  organization  or  product  are  prepared  by  the 
organization's  sub-units. 

C.  No  written  plan(s)  are  prepared  that  set  long-term  product  or 
organizational  goals,  standards,  or  objectives. 
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7.  Considering  all  of  the  goals,  standards,  or  objectives  contained 
in  your  plan(s),  how  many  are  written  in  terms  that  are  measurable 
or  quantifiable?  (CHECK  ONE.) 

A.  All 

B .  Most 

C.  About  half 

D .  Some 

E.  Few  or  none 

8.  Considering  all  of  the  goals,  standards,  or  objectives  contained 
in  your  long-term  plan(s) ,  how  many  are  translated  into  annual  or 
semi-annual  operating  plans?  (CHECK  ONE.) 

A.  All 

B .  Most 

C.  About  half 

D .  Some 

E.  Few  or  none 

9.  How  many  of  your  organization's  goals,  standards,  or  objectives 
included  in  your  plan(s)  have  related  measures  of  progress?  (CHECK 
ONE.  ) 

A.  All 

B.  Most 

C.  About  half 

D .  Some 


E.  Few  or  none 
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10.  Are  achievement  of  goals,  standards,  and  objectives  in  your 
organization's  plan (s) reflected  in  senior  management  performance 
appraisals? 

A.  All 

B .  Most 

C.  About  half 

D .  Some 

E.  Few  or  none 


11.  Are  organizational  performance  measures  reflected  in  senior 
management  performance  appraisals? 

A.  All 

B.  Most 

C.  About  half 

D .  Some 

E.  Few  or  none 


12.  Are  achievement  of  Total  Quality  objectives  reflected  in  senior 
management  performance  appraisals? 

A.  All 

B .  Most 

C.  About  half 

D .  Some 

E.  Few  or  none 
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13.  Does  your  organization  relate  the  following  to  employee  or 
management  incentive  plans?  (CHECK  ALL  THAT  APPLY.) 

A.  Goals,  standards,  and  objectives  in  your  organization's  plan(s) 

B .  Performance  measures 

C.  Total  Quality  objectives 


14.  Is  there  a  central  office  in  your  organization  that  is 
responsible  for  managing,  monitoring,  collecting,  or  internally 
distributing  all  or  most  performance  measures  for  the  entire 
organization?  (CHECK  ONE) 

A.  No 

B.  Yes--  Please  provide  the  following: 

Name  of  Central  Office: 

Person  Responsible: 

Title: 

Phone  Number: 


IV.  REPORTING  AND  USE  OF  MEASURES 


15.  Are  current  performance  measures  distributed  internally  as  a 
single  report  to  top  organization  management?  (CHECK  ONE.) 

A.  Yes  --  How  many  times  per  year  is  the 
report  issued? 

_  Times  per  year 

B.  No,  there  are  multiple  reports 

C.  No,  there  are  no  reports 


16.  What  key  performance  measurement  indicators  and  ratios  does  your 
organization  use?  e.g.  number  of  errors  per  volume,  projects 
completed  per  time  elapsed,  net  cost/patient,  etc.  (If  possible, 
please  attach  copies  of  your  organization's  indicators  and  ratios. 
Confidentiality  is  guaranteed.) 


92 


17.  Does  your  organization  benchmark?  (Benchmarking  is  defined  as 
searching  for  the  best  practices  that  will  help  define  superior 
products,  services,  or  support  processes.) 

A.  Yes  - >  (CONTINUE  WITH  QUESTION  16) 

B.  No  - >  (SKIP  TO  QUESTION  18) 


18.  Does  your  organization  benchmark  the  following? 

A.  Indicators  and  Ratios 

B.  Methods  or  Processes 

C.  Both 


19.  To  what  does  your  organization  compare  its  performance  measures? 

(CHECK  ALL  THAT  APPLY.) 

A.  To  historical  trends  within  the  same  product  or 
service  line . 

B.  To  product  or  service  goals,  standards,  or 
objectives  (e.g.  in  annual  operating  or  strategic 
plans . ) 

C.  To  other  units  or  regions  within  the  MEDCOM. 

D.  To  alternative  suppliers  or  similar  operations 
(e.g.,  other  companies  or  competitors.) 

E.  To  the  best  domestic  organizations. 

F.  To  world  class  organizations. 

G.  Other  (Please  specify) 

H.  No  comparisons  are  made 
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20.  Does  your  organization  use  automated  measurement  information 
systems? 

A.  Yes  -  If  so,  is  the  system: 

An  off  the  shelf  variety? 

Custom  designed? 

Military  procured? 

B.  No 

V.  SATISFACTION  WITH  USE  OF  PERFORMANCE  MEASURES  IN  MANAGEMENT 
PROCESS 


21.  How  satisfied  or  dissatisfied  is  your  activity  with  your  current 
performance  measures  as  a  tool  to  help  your  organization  do  the 
following? 

Very  Somewhat  Neither  Somewhat  Very 

Satisfied  Satisfied  Satisfied  Nor  Dissatisfied  Dissatisfied 

Dissatisfied 


Make  budget 
decisions 

Manage  products 
or  services 

Assure 

accountability 

Measure  product/ 
service  results 
or  outcomes 
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22.  Overall,  how  satisfied  or  dissatisfied  is  your  activity  with  the 
timeliness,  reliability,  and  level  of  detail  of  the  information 
produced  by  your  organization's  current  performance  measures? 

Very  Somewhat  Neither  Somewhat  Very 

Satisfied  Satisfied  Satisfied  Nor  Dissatisfied  Dissatisfied 

Dissatisfied 

Timeliness  of  the 
information  produced 
by  the  measures 

Reliability  of  the 
information  produced 
by  the  measures 

Level  of  detail  of 
the  information 
produced  by  the 
measures 
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VI.  APPLICABILITY  OF  PERFORMANCE  MEASURES  TO  MEDCOM 


23.  Please  indicate  the  types  of  performance  measures  that  you  would 
recommend  be  used  for  the  following  purposes: 

Measure  Manage  Assure  Make 

Products/  Products/  Account-  Budget 

Services  Services  ability  Decisions 


Performance  Measures : 

Resource  Inputs 

Work/Activity 

Level 

Complexity  of  Work 
Processes 

Internal  Measures 
of  Quality 

External  Customer 
Needs 

External  Measures  of 
Customer  Satisfaction 

Timeliness  of 
Product  or  Service 

Outputs  or  Final 
Products 

Financial  Indicators 
Efficiency  Measures 


24.  Do  you  think  the  MEDCOM  should  re-engineer  the  current 
performance  measurement  system? 

A.  Yes 

B.  No 


25.  Do  you  have  any  other  recommendations  for  implementing 
performance  measurement  in  the  MEDCOM? 


26.  Please  provide  any  additional  comments  you  may  have. 


Upon  Completion,  please  mail  to 


Headquarters ,  USAMEDCOM 
ATTN : MCCG- PAE  (CPT  Swanson) 
2050  Worth  Road 
Fort  Sam  Houston,  TX  7823-6000 

or  FAX  to: 

Commercial : (210)  221-7901 
DSN:  471-7901 
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